Detecting Duplicate Messages

A guaranteed message contains a replication group message ID. This message ID can be used to detect duplicate message delivery, so the application can attempt to avoid processing the same message more than once.

To Get a Message's Replication Group Message ID

PubSub+ Messaging API	Use
JCSMP	XMLMessage.getReplicationGroupMessageId()
Java RTO	MessageHandle.getReplicationGroupMessageIdForHandle(…)
C	solClient_msg_getReplicationGroupMessageId(…)
.NET	IMessage.ReplicationGroupMessageId
Javascript and Node.js	Message.getReplicationGroupMessageId()

The ID for a particular message is only guaranteed to be the same for a particular copy of a message on a particular queue or topic endpoint within a replication group. The same message on different queues or topic endpoints within the same replication group may or may not have the same replication group message ID. The same message propagated to different replication groups via bridges, DMR links, MNR links, or multiple publishes by the application will have different replication group message ID’s. So, the ID's intended use is for an application consuming messages from a particular queue or topic endpoint within a replication group to avoid processing the same message from that queue or topic endpoint more than once. Note that if a message is moved from a queue or topic endpoint to its associated dead message queue, the message will get a different replication group message ID.

The following techniques can be used to detect duplicate messages, each having their own pros and cons. The technique you decide to use will depend on the nature of your application.

• Looking up Replication Group Message IDs in a Database

• Comparing Replication Group Message IDs

Both techniques require converting a replication group message ID to a string, which can be persistently stored by the application.

To Convert a Replication Group Message ID to a String

PubSub+ Messaging API	Use
JCSMP	ReplicationGroupMessageId.toString()
Java RTO	ReplicationGroupMesageIdHandle.toString()
C	solClient_replicationGroupMessageId_toString(…)
.NET	IReplicationGroupMessageId.ToString()
Javascript and Node.js	ReplicationGroupMessageId.toString()

Looking up Replication Group Message IDs in a Database

This technique is only practical for applications that maintain a history of messages processed in a database. An application using this technique would follow these steps:

1. Receive a message.

2. Convert the message’s replication group message ID to a string.

3. Lookup the string form of the replication group message ID in the database.

4. If found, the message has already been processed. Acknowledge the message to delete it from the queue.

5. If not found, process the message, and store the string in a database, ensuring this is all done as an atomic transaction.

Comparing Replication Group Message IDs

This technique requires that only the last message ID processed is stored. However, there are cases where the comparison between the two ID’s can fail. Failures should be considered rare, though they can occur if the messages being compared originated from different replication sites. On the other hand, these failures are useful in detecting duplicates after a replication failover because the greatest chance of duplicates after a failover involve messages that originated from the mate site of the newly active replication site.

Since the comparison of replication group message ID objects can fail, the object’s comparison function intentionally does not implement standard comparison interfaces present in some of API’s native languages. These interfaces assume the only cause of failure is due to an application error, and do not allow for the specification of checked exceptions that are intended to be handled by the application.

Where this technique stands the greatest chance of failing to detect a duplicate is if a replication failover occurs before a client has consumed all messages originated from the mate replication site from the previous replication failover.

An application using this technique would follow these steps:

1. On start up, the application would retrieve the most recently processed message’s replication group message ID from persistent storage, and convert this string into the application’s most recently processed replication group message ID object. This string must be equal to the string previously obtained by using the API to convert a message’s replication group message ID to a string.

2. Receive a message.

3. Retrieve the message’s replication group message ID.

4. Compare the message’s replication group message ID to the most recently processed message.

5. If the most recently processed message is not newer, discard the message as a duplicate.

6. If the comparison fails, the application must decide how to proceed. Most applications would want to ensure there is no chance of losing a message and would wish to treat the message as “unprocessed” and so it would handle the message as if it were a newer message.

7. If the most recently processed message is newer, the application would convert the message’s replication group message ID to a string, then process the message and store the string persistently as the most recently processed message as a transaction. This persistent store would be read on the next application start up as described in step 1. Then the application’s most recently processed replication group message ID is updated to be this message’s replication group message ID object before receiving the next message.

To Convert a String into a Replication Group Message ID

PubSub+ Messaging API	Use
JCSMP	JCSMPFactory.createReplicationGroupMessageId(…)
Java RTO	Solclient.createReplicationGroupMessageIdForHandle(…)
C	solClient_replicationGroupMessageId_fromString(…)
.NET	ContextFactory.CreateReplicationGroupMessageId(…)
Javascript and Node.js	SolclientFactory.createReplicationGroupMessageId(…)

To Compare a Replication Group Message ID to Another Replication Group Message ID

PubSub+ Messaging API	Use
JCSMP	ReplicationGroupMessageId.compare(…)
Java RTO	ReplicationGroupMesageIdHandle.compare(…)
C	solClient_replicationGroupMessageId_compare(…)
.NET	IReplicationGroupMessageId.Compare(…)
Javascript and Node.js	ReplicationGroupMessageId.compare(…)