How Replication Works

This section provides a simplified description of how replication works. All diagrams and descriptions are for the synchronous mode of replication, where messages are stored on both sites before responding to the client. For more details about message replication, see Synchronous and Asynchronous Message Replication.

The process of replication can differ depending on whether the guaranteed messages are part of a transaction. There are also differences in how local and XA transactions are handled.

Non-Transacted Messages

The following diagram provides an example of how replication propagates non-transacted Guaranteed messages that have been published to a topic. The diagram shows a replication group consisting of the New York data center (the active replication site), and the Boston data center (the standby replication site).

The detailed steps of this process flow (corresponding to the numbers in the diagram) are as follows:

  1. The client publishes a message that matches a replicated topic subscription that is configured for the Message VPN with an active replication state. The message is persisted on the event broker.
  2. A copy of the message is sent over a replication bridge (named #MSGVPN_REPLICATION_BRIDGE) that is automatically created when replication is enabled for a Message VPN. This bridge links the Message VPN with the active replication state in the New York data center and its mate Message VPN of the same name that has a standby replication state in the Boston datacenter.

    3. For added security, Transport Layer Security (TLS) / Secure Sockets Layer (SSL) encryption can be used on Message VPN replication bridges and replication Config-Sync bridges. For information on using SSL encryption on replication Config-Sync bridges and Message VPN replication bridges, refer to TLS / SSL Service Configuration.

  3. The message is also persisted on the replication mate.
  4. When the message is consumed from an endpoint on the replication active Message VPN, it is removed from the persistent store.
  5. An acknowledgment (ACK) is propagated to the replication standby Message VPN that the message has been delivered to the consumer.
  6. The message is then also removed from the persistent store of the event broker the replication standby Message VPN resides on.

Consumer ACK Propagation

After a client consumes a message from a durable endpoint, an acknowledgment (that is, an ‘ACK’) is generated so that the message is removed from the message spool. (An ACK is also generated if a message is deleted from the message spool.)

By default, for a Message VPN with an active replication state, after clients consume a set number of messages from its endpoints, an ACK message is propagated to its mate Message VPN with a standby replication state. When this ACK message is received by the corresponding durable endpoints on the mate replication Message VPN, those endpoints remove those same messages that were replicated to it. This process ensures that the message states remain synchronized between both sites.

There is a configurable interval for propagating ACK messages, which affects bandwidth utilization on the replication bridge. See Configuring ACK Propagation Intervals.

If you have a dead message queue (DMQ) configured for an endpoint, disabling ACK propagation for that endpoint means that the time-to-live (TTL) timers will run independently at both sites, potentially leading to differences in spooled messages between sites. For more information, see Dead Message Queues.

Consumer ACK Propagation and Non-Replicated Messages

If you have an endpoint with consumer ACK propagation enabled that contains non-replicated messages, those messages will be deleted in the case of a replication failover. This is because consumer ACKs on the newly active site are propagated to the new standby, triggering message deletion.

If you want non-replicated messages on certain endpoints to survive a replication failover, consider disabling consumer ACK propagation on those endpoints. Depending on your use case, you may also want to enable allow-clients-when-standby so that non-replicated messages can be drained from those endpoints.

Local Transactions

With a local transaction, the messages and acknowledgments are propagated to the replication mate as a part of an internal XA transaction.

For publishing messages in a local transaction, the process flow is as follows:

  1. The client opens a local transaction and publishes multiple messages that match a replicated topic subscription. The messages are persisted on the active site.
  2. The client commits the transaction. An internal XA transaction containing the messages is created.
  3. Copies of the messages are sent over the replication bridge to the standby site as part of the internal XA transaction and persisted on the standby site.
  4. The internal XA transaction is committed on the standby site. The messages are inserted into the destination endpoint(s) on the standby site.
  5. The messages are inserted into the destination endpoints on the active site. The internal XA transaction completes. The commit response is sent back to the client.

For consuming messages, the process is as follows:

  1. The client opens a local transaction and consumes multiple messages that have been replicated (that is, when they were published, they matched a replicated topic subscription) from an endpoint.
  2. The client commits the transaction. An internal XA transaction containing an acknowledgment of the consumed messages is created.
  3. The acknowledgment is sent over the replication bridge to standby site as part of the internal XA transaction.
  4. The internal XA transaction is committed on the standby site. The consumed messages are removed from the endpoint on the standby site.
  5. The internal XA transaction completes. The consumed messages are removed from the endpoint on the active site. The commit response is sent back to the client.

It is possible for a single local transaction to include both the publishing and consuming of messages. For simplicity, the examples show publishing and consuming in separate transactions.

XA Transactions

With an XA transaction, the standby site is included in the distributed transaction.

This is the process for publishing messages in an XA transaction:

  1. The client starts an XA transaction and publishes multiple messages that match a replicated topic subscription. The messages are persisted on the active site.
  2. Copies of the messages are sent immediately over the replication bridge to the standby replication site. The messages are persisted on the standby site.
  3. The client ends the XA transaction. The transaction state is updated on the active site.
  4. The client prepares the XA transaction. The active site checks if the transaction can succeed. If it can, the prepare request is sent to the standby site, where it is checked. If it can also succeed on the standby site, the XA transaction state is updated on both active and standby sites. The prepare response is sent back to the client.
  5. The client commits the XA transaction. The messages are inserted into the destination endpoints on the active site. The commit request is sent to the standby site and the messages are inserted into the destination endpoints on the standby site.
  6. The commit response is sent back to the client.
  7. The XA transaction is completed.

This is the process for consuming messages in an XA transaction:

  1. The client starts an XA transaction and consumes multiple messages that has been replicated (that is, when they were published, they matched a replicated topic subscription).
  2. The client ends the XA transaction. The transaction state is updated with the acknowledgment on the active site.
  3. The client prepares the XA transaction. The active site checks if the transaction can succeed. If it can, the prepare request is sent to the standby site, where it is checked. If it can also succeed on the standby site, the XA transaction state is updated on both active and standby sites. The prepare response is sent back to the client.
  4. The client commits the XA transaction. The messages are removed from endpoint on the active site. The commit request is sent to the standby site and the messages are removed from the endpoint on the standby site.
  5. The commit response is sent back to the client.
  6. The XA transaction is completed.

It is possible for an XA transaction to include both the publishing and consuming of messages. For simplicity, the examples show publishing and consuming in separate transactions.