Configuring Replication with a DMR Network

You can take advantage of disaster recovery (using replication and Config-Sync) while using Dynamic Message Routing (DMR) to dynamically route messages through your event mesh. DMR provides a means to interconnect instances of Solace PubSub+ event brokers so that messages published by clients hosted by one broker can be consumed by clients hosted on another. After the relationships between brokers are established, the broker automatically discovers how to route messages. Client applications don't need to take any action to publish or consume messages across the event mesh.

For more information, see the following:

In version 9.6, the internal naming of channels and queues changed to support replication with DMR. Brokers older than version 9.6 cannot learn the new naming, so cluster links between a version 9.6 broker and an older broker will fail.

Do not create any DMR cluster links (in either direction) between a version 9.6 broker and an older broker.

If you are upgrading brokers from version 9.5 or older that are part of a DMR network, do not add any new DMR cluster links anywhere in the network until all brokers in the network have been upgraded to version 9.6 or newer. Links that existed before the upgrade will continue to function properly.

Using These Procedures

The instructions that follow are applicable for both appliances and software event brokers, and for both single brokers and HA pairs.

Where a broker A (or B, etc.) is referenced, it may have an HA mate A' (or B', etc.). These procedures assume the following:

  • broker A may have an HA mate A'
  • A is the primary broker and A' is the backup
  • the redundancy status is UP
  • Config-Sync is enabled, and A and A' are currently synchronized

So long as Config-Sync is operational, you need to make configuration changes only on broker A of the A/A' pair. Any special considerations required are listed with each procedure.

Failing to follow these procedures, or using a procedure not listed here, may result in message loss.

Adding a Replication Mate to a Node in a DMR Network

You can add a replication mate to a node in a DMR network without causing a service interruption to existing clients. During the procedure it is expected that the state of the cluster will be reported as DOWN and topology errors will be raised while links are being established between nodes. In spite of these errors, traffic will continue to flow through the network and no message loss is expected.

Assumptions

  • broker A in cluster C has existing DMR links (internal or external) to brokers X1 through Xn
  • the state of the clusters on all of A and X1..n is UP
  • broker B is to be added as the replication mate of A
  • all brokers using HA or replication have Config-Sync enabled and UP

High Availability Considerations

For software event brokers, you may have to provide a second connect-via address when you configure replication and links in the steps that follow. For example, when you configure the cluster link on A to B/B', you may have to provide the addresses for both B and B'.

Steps

  1. Configure replication as normal between A and B, such that A is active and B standby:

    1. Set the global replication attributes on both A and B, such that A is the replication mate of B and vice-versa. See Configuring System-Level Replication Settings.
    2. Create all Message VPNs on B that will be replicated from A. See Configuring Message VPNs.
    3. Enable replication for Message VPNs of A (active) and B (standby). See Enabling Replication.

      On both A and B, every Message VPN for which DMR is enabled must also have replication enabled.

  2. Wait for Config-Sync on A and B to report all replicated Message VPNs as being Ready-InSync.
  3. Create cluster C on broker B, and configure its authentication and other attributes as needed. See Configuring a DMR Cluster.
  4. Create a new cluster link on A to B and on B to A (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via must NOT be set
    • enabled
  5. Verify that the link between A and B is UP. See Displaying Cluster Link Information.

  6. For each broker X1 through Xn do the following:
    1. Wait for the cluster on X to report a missing link between X and B.
    2. Create a new cluster link on X to B and on B to X (see Configuring Cluster Links). The new link attributes must match the existing ones to and from A, other than those that relate specifically to B:
      • authentication settings
      • connect-via
    3. If a new link has a span of external, on X, create the DMR bridges for each VPN between X and B, identical to those between X and A. See Configuring a DMR Bridge.
  7. Verify that the links between X1 through Xn and B are UP. See Displaying Cluster Link Information.

  8. Verify that the cluster is UP on all brokers. See Monitoring Your DMR Cluster.

The new replication mate B is now fully integrated into the DMR network.

Optional: You can now move selected replicated Message VPNs on A such that they are active on B.

Removing a Replication Mate from a Node in a DMR Network

You can remove a replication mate from a node in a DMR network without causing a service interruption to existing clients if the node being removed is not hosting any clients (i.e. it is replication standby). During the procedure it is expected that the state of the cluster will be reported as DOWN and topology errors will be raised while links are being removed between nodes. In spite of these errors, traffic will continue to flow through the network and no message loss is expected.

Assumptions

  • broker A in cluster C has existing DMR links (internal or external) to brokers X1 through Xn
  • the state of the cluster(s) on broker A and all brokers X1..n is UP
  • broker B (to be removed) is the replication mate of A
  • all Message VPNs have an active replication state on broker A and a standby replication state on broker B
  • all brokers using redundancy or replication have Config-Sync enabled and UP

Steps

  1. On each broker X1 through Xn, disable and remove the cluster link to B:
    broker-X1(configure)# routing
    broker-X1(configure/routing)# dynamic-message-routing
    broker-X1(...igure/routing/dynamic-message-routing)# cluster cluster-C
    broker-X1(...uting/dynamic-message-routing/cluster)# link broker-B
    broker-X1(...uting/dynamic-message-routing/cluster)# shutdown
    broker-X1(...uting/dynamic-message-routing/cluster)# exit
    broker-X1(...uting/dynamic-message-routing/cluster)# no link broker-B
    
    broker-X2(configure)# routing
    broker-X2(configure/routing)# dynamic-message-routing
    broker-X2(...igure/routing/dynamic-message-routing)# cluster cluster-C
    broker-X2(...uting/dynamic-message-routing/cluster)# link broker-B
    broker-X2(...uting/dynamic-message-routing/cluster)# shutdown
    broker-X2(...uting/dynamic-message-routing/cluster)# exit
    broker-X2(...uting/dynamic-message-routing/cluster)# no link broker-B
    
    ...

    The resulting cluster looks like the following:

  2. Disable and remove cluster C on B:
    broker-B(configure)# routing
    broker-B(configure/routing)# dynamic-message-routing
    broker-B(...igure/routing/dynamic-message-routing)# cluster cluster-C
    broker-B(...igure/routing/dynamic-message-routing)# shutdown
    broker-B(...igure/routing/dynamic-message-routing)# exit
    broker-B(...igure/routing/dynamic-message-routing)# no cluster cluster-C
    
  3. Disable and remove the cluster link on A to B (as in step 1).

    Node B is no longer part of the cluster, but remains the replication mate of A, as shown in the following diagram:

  4. If a topology issue One-way link between B and A is present on A, wait for it to clear. This may take several minutes.

    Proceeding to the next step before this topology issue clears may result in loss of persistent messages. Note that the presence of topology issue External link missing between B and X is normal at this step.

  5. Disable and remove replication as normal between A and B.
    1. Disable replication for all Message VPNs of both A and B:
      broker-A(configure)# message-vpn vpn-one
      broker-A(configure/message-vpn)# replication shutdown
      broker-A(configure/message-vpn)# exit
      broker-A(configure)# message-vpn vpn-two
      broker-A(configure/message-vpn)# replication shutdown
      broker-A(configure/message-vpn)# exit
      ...
      broker-B(configure)# message-vpn vpn-one
      broker-B(configure/message-vpn)# replication shutdown
      broker-B(configure/message-vpn)# exit
      broker-B(configure)# message-vpn vpn-two
      broker-B(configure/message-vpn)# replication shutdown
      broker-B(configure/message-vpn)# exit
      ...
    2. Remove global replication attributes on both A and B:
      broker-A(configure)# replication
      broker-A(configure/replication)# no interface
      broker-A(configure/replication)# no mate broker-B
      broker-A(configure/replication)# no mate connect-via <addr-port>
      broker-B(configure)# replication
      broker-B(configure/replication)# no interface
      broker-B(configure/replication)# no mate broker-A
      broker-B(configure/replication)# no mate connect-via <addr-port>
      

      In the preceding code, <addr-port> is the IP address or hostname of the event broker in the form IP-address:port or hostname:port, where the IP address is specified in the dotted decimal notation form nnn.nnn.nnn.nnn and port is the port number on the replication mate virtual router (specified as a decimal value from 0 to 65535).

  6. Verify that the cluster is UP on all brokers. See Monitoring Your DMR Cluster.

At this point B is fully removed from the DMR network and is no longer the replication mate of A, as shown below:

Pairing Two Nodes for Replication in a DMR Cluster

It is service interrupting to make two nodes already part of a DMR network into a replication group—any clients that were hosted by one broker must be rehosted on the other. During the procedure it is expected that the state of the cluster will be reported as DOWN and topology errors will be raised for a period of time as links are being adjusted between nodes. Regardless of these errors, traffic will continue to flow through the network.

On one of the brokers to be paired (in this case, B), messages that are either destined for or currently enqueued on that broker will be lost.

Assumptions

  • brokers A and B are both in cluster C, joined by an internal link
  • brokers A and B have existing DMR links (internal or external) to brokers X1 through Xn
  • the state of the clusters on all of A, B, and X1..n is UP
  • broker B is to become the replication mate of A
  • all brokers using HA or replication have Config-Sync enabled and UP (however, if B is a non-redundant broker Config-Sync is operationally DOWN on B until step 7, below)
  • clients of the replication group A/B have been configured to include the addresses of both A and B in their host lists

High Availability Considerations

  • For software event brokers, you may have to provide a second connect-via address when you configure replication and links in the steps that follow. For example, when you configure the cluster link on A to B/B', you may have to provide the addresses for both B and B'.
  • Enabling/disabling of the msg-backbone must be done on both B and B'.

Steps

  1. Shut down the msg-backbone service on B. Any clients connected to B will move to A.
    broker-B(configure)# service msg-backbone shutdown
  2. Shut down and remove all cluster links on B to brokers X1 through Xn
    broker-B(configure)# routing
    broker-B(configure/routing)# dynamic-message-routing
    broker-B(...igure/routing/dynamic-message-routing)# cluster cluster-C
    broker-B(...uting/dynamic-message-routing/cluster)# link broker-X1
    broker-B(...uting/dynamic-message-routing/cluster)# shutdown
    broker-B(...uting/dynamic-message-routing/cluster)# exit
    broker-B(...uting/dynamic-message-routing/cluster)# no link broker-X1
    broker-B(...uting/dynamic-message-routing/cluster)# link broker-X2
    broker-B(...uting/dynamic-message-routing/cluster)# shutdown
    broker-B(...uting/dynamic-message-routing/cluster)# exit
    broker-B(...uting/dynamic-message-routing/cluster)# no link broker-X2
    ...
    broker-B(...uting/dynamic-message-routing/cluster)# link broker-Xn
    broker-B(...uting/dynamic-message-routing/cluster)# shutdown
    broker-B(...uting/dynamic-message-routing/cluster)# exit
    broker-B(...uting/dynamic-message-routing/cluster)# no link broker-Xn
  3. On each broker X1 through Xn, disable and remove the cluster link to B.
    broker-X1(configure)# routing
    broker-X1(configure/routing)# dynamic-message-routing
    broker-X1(...igure/routing/dynamic-message-routing)# cluster <cluster-name>
    broker-X1(...uting/dynamic-message-routing/cluster)# link broker-B
    broker-X1(...uting/dynamic-message-routing/cluster)# shutdown
    broker-X1(...uting/dynamic-message-routing/cluster)# exit
    broker-X1(...uting/dynamic-message-routing/cluster)# no link broker-B
    broker-X2(configure)# routing
    broker-X2(configure/routing)# dynamic-message-routing
    broker-X2(...igure/routing/dynamic-message-routing)# cluster <cluster-name>
    broker-X2(...uting/dynamic-message-routing/cluster)# link broker-B
    broker-X2(...uting/dynamic-message-routing/cluster)# shutdown
    broker-X2(...uting/dynamic-message-routing/cluster)# exit
    broker-X2(...uting/dynamic-message-routing/cluster)# no link broker-B
    ...

    Node B is now connected only to Node A, as shown below:

  4. Configure replication as normal between A and B, such that A is active and B standby:
    1. Set global replication attributes on both A and B, such that A is the replication mate of B and vice-versa. See Configuring Replication Mates.
    2. Create all Message VPNs on B that will be replicated from A. See Configuring Message VPNs.
    3. Enable replication for Message VPNs of A (active) and B (standby). See Enabling Replication.

      On both A and B, every Message VPN for which DMR is enabled must also have replication enabled.

  5. Remove the connect-via settings of the cluster link on A to B and on B to A.
    broker-A(configure)# routing
    broker-A(configure/routing)# dynamic-message-routing
    broker-A(...igure/routing/dynamic-message-routing)# cluster cluster-C
    broker-A(...uting/dynamic-message-routing/cluster)# link broker-B
    broker-A(.../dynamic-message-routing/cluster/link)# shutdown
    broker-A(.../dynamic-message-routing/cluster/link)# no connect-via <addr-port>
    …
    broker-A(...uting/dynamic-message-routing/cluster)# link broker-B
    broker-A(.../dynamic-message-routing/cluster/link)# no shutdown
    broker-B(configure)# routing
    broker-B(configure/routing)# dynamic-message-routing
    broker-B(...igure/routing/dynamic-message-routing)# cluster cluster-C
    broker-B(...uting/dynamic-message-routing/cluster)# link broker-A
    broker-B(.../dynamic-message-routing/cluster/link)# shutdown
    broker-B(.../dynamic-message-routing/cluster/link)# no connect-via <addr-port>
    …
    broker-B(...uting/dynamic-message-routing/cluster)# link broker-A
    broker-B(.../dynamic-message-routing/cluster/link)# no shutdown
    
    

    In the preceding code, <addr-port> is the IP address or hostname of the event broker in the form IP-address:port or hostname:port, where the IP address is specified in the dotted decimal notation form nnn.nnn.nnn.nnn and port is the port number on the replication mate virtual router (specified as a decimal value from 0 to 65535).

  6. Enable the msg-backbone service on B.
    broker-B(configure)# no service msg-backbone shutdown
  7. Wait for Config-Sync on A and B to report all replicated Message VPNs as being in-sync.
  8. Verify that the link between A and B is UP. See Displaying Cluster Link Information.
  9. For each broker X1 through Xn do the following:
    1. Wait for the cluster on X to report a missing link between X and B.
    2. Create a new link on X to B and on B to X (see Configuring Cluster Links). At a minimum, configure these link attributes:
      • span to either internal or external as appropriate
      • authentication settings
      • connect-via
      • enabled

  10. Verify that the links between X1 through Xn and B are UP. See Displaying Cluster Link Information.
  11. Verify that the cluster is UP on all brokers. See Monitoring Your DMR Cluster.

The new replication mate B is now fully integrated into the DMR network.

Optional: You can now move selected replicated Message VPNs on A such that they are active on B.

Adding a Replication Group to a DMR Cluster

You can add an existing replication group as new nodes to a DMR network without causing a service interruption to existing clients. During the procedure it is expected that the state of the cluster will report as being DOWN and topology errors will be raised for a period of time as links are being added between nodes. Regardless of this, existing traffic flow through the network will continue, and no message loss is expected.

Assumptions

  • brokers A and B are a replication group without a cluster configured
  • all replicated Message VPNs on A and B are active on A and standby on B
  • all Message VPNs on X1 through Xn for which DMR is enabled exist and are replicated on A and B
  • brokers X1 through Xn belong to cluster C
  • the state of the cluster on all of X1..n is UP
  • all brokers using HA or replication have Config-Sync enabled and UP

High Availability Considerations

For software event brokers, you may have to provide a second connect-via address when you configure replication and links in the steps that follow. For example, when you configure the cluster link on A to B/B', you may have to provide the addresses for both B and B'.

Steps

  1. Create cluster C on brokers A and B, and configure its authentication and other attributes as needed. See Configuring a DMR Cluster.

  2. On A, enable DMR for every Message VPN on X1 through Xn for which DMR is enabled.
    broker-A(configure)# message-vpn vpn-one
    broker-A(configure/message-vpn)# dynamic-message-routing
    broker-A(...e/message-vpn/dynamic-message-routing)# no shutdown
    broker-A(configure/message-vpn)# exit
    broker-A(configure)# message-vpn vpn-two
    broker-A(configure/message-vpn)# dynamic-message-routing
    broker-A(...e/message-vpn/dynamic-message-routing)# no shutdown
    broker-A(configure/message-vpn)# exit
    ...
  3. Create new cluster links from A to B and from B to A (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via must NOT be set
    • enabled

  4. Verify that the link between A and B is UP (see Displaying Cluster Link Information), and that cluster C is UP on both A and B (see Monitoring Your DMR Cluster.).
  5. Create a new cluster link on A to each broker X1 through Xn (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via
    • enabled
  6. On each broker X1 through Xn, create a new link to A (see Configuring Cluster Links). At a minimum, configure these link attributes: :
    • span to be internal
    • authentication settings
    • connect-via
    • enabled
  7. Verify that the links between X1 through Xn and A are UP (see Displaying Cluster Link Information).

  8. Configure a new link on B to each broker X1 through Xn (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via
    • enabled
  9. For each broker X1 through Xn do the following:
    1. Wait for the cluster on X to report a missing link between X and B.
    2. Create a new link on X to B (see Configuring Cluster Links). At a minimum, configure these link attributes:
      • span to be internal
      • authentication settings
      • connect-via
      • enabled
  10. Verify that the cluster is UP on all brokers. See Monitoring Your DMR Cluster.

At this point the new replication group A/B is fully integrated into the DMR network. If desired, selected replicated Message VPNs on A could be moved such that they are active on B.