Configuring Replication with a DMR Network

You can take advantage of disaster recovery (using replication and Config-Sync) while using Dynamic Message Routing (DMR) to dynamically route messages through your event mesh. DMR provides a means to interconnect instances of Solace PubSub+ event brokers so that messages published by clients hosted by one broker can be consumed by clients hosted on another. After the relationships between brokers are established, the broker automatically discovers how to route messages. Client applications don't need to take any action to publish or consume messages across the event mesh.

For more information, see the following:

Using These Procedures

The instructions that follow are applicable for both appliances and software event brokers, and for both single brokers and HA pairs.

Where a broker A (or B, etc.) is referenced, it may have an HA mate A' (or B', etc.). These procedures assume the following:

  • broker A may have an HA mate A'
  • A is the primary broker and A' is the backup
  • the redundancy status is UP
  • Config-Sync is enabled, and A and A' are currently synchronized
  • all nodes in the network are at the same software level

In versions earlier than 10.0.1 replication was required to be enabled on all DMR-enabled Message VPNs. In versions 10.0.1 and later, replication may be enabled or disabled on all DMR-enabled Message VPNs.

So long as Config-Sync is operational, you need to make configuration changes only on broker A of the A/A' pair. Any special considerations required are listed with each procedure.

Failing to follow these procedures, or using a procedure not listed here, may result in message loss.

Adding a Replication Mate to a Node in a DMR Network

You can add a replication mate to a node in a DMR network without causing a service interruption to existing clients. During the procedure it is expected that the state of the cluster will be reported as DOWN and topology errors will be raised while links are being established between nodes. In spite of these errors, traffic will continue to flow through the network and no message loss is expected.

Assumptions

  • broker A in cluster C has existing DMR links (internal or external) to brokers X1 through Xn
  • the state of the clusters on all of A and X1..n is UP
  • broker B is to be added as the replication mate of A
  • all brokers using HA or replication have Config-Sync enabled and UP

High Availability Considerations

For software event brokers, you may have to provide a second connect-via address when you configure replication and links in the steps that follow. For example, when you configure the cluster link on A to B/B', you may have to provide the addresses for both B and B'.

Steps

  1. Configure replication as normal between A and B, such that A is active and B standby:

    1. Set the global replication attributes on both A and B, such that A is the replication mate of B and vice-versa. See Configuring System-Level Replication Settings.
    2. Create all Message VPNs on B that will be replicated from A. See Configuring Message VPNs.
    3. Enable replication for Message VPNs of A (active) and B (standby). See Enabling Replication.

  2. Wait for Config-Sync on A and B to report all replicated Message VPNs as being Ready-InSync.
  3. Create cluster C on broker B, and configure its authentication and other attributes as needed. See Configuring a DMR Cluster.
  4. Create a new cluster link on A to B and on B to A (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via must NOT be set
    • enabled

  5. Verify that the link between A and B is UP. See Displaying Cluster Link Information.

  6. For each broker X1 through Xn do the following:
    1. Wait for the cluster on X to report a missing link between X and B.
    2. Create a new cluster link on X to B and on B to X (see Configuring Cluster Links). The new link attributes must match the existing ones to and from A, other than those that relate specifically to B:
      • authentication settings
      • connect-via
    3. If a new link has a span of external, on X, create the DMR bridges for each VPN between X and B, identical to those between X and A. See Configuring a DMR Bridge.

  7. Verify that the links between X1 through Xn and B are UP. See Displaying Cluster Link Information.

  8. Verify that the cluster is UP on all brokers. See Displaying Cluster Information.

The new replication mate B is now fully integrated into the DMR network.

Optional: You can now move selected replicated Message VPNs on A such that they are active on B.

Removing a Replication Mate from a Node in a DMR Network

You can remove a replication mate from a node in a DMR network without causing a service interruption to existing clients if the node being removed is not hosting any clients (i.e. it is replication standby). During the procedure it is expected that the state of the cluster will be reported as DOWN and topology errors will be raised while links are being removed between nodes. In spite of these errors, traffic will continue to flow through the network and no message loss is expected.

Assumptions

  • broker A in cluster C has existing DMR links (internal or external) to brokers X1 through Xn
  • the state of the cluster(s) on broker A and all brokers X1..n is UP
  • broker B (to be removed) is the replication mate of A
  • all Message VPNs have an active replication state on broker A and a standby replication state on broker B
  • all brokers using redundancy or replication have Config-Sync enabled and UP

Steps

  1. On each broker X1 through Xn, disable and remove the cluster link to B:
    broker-X1(configure)# routing
broker-X1(configure/routing)# dynamic-message-routing
broker-X1(...igure/routing/dynamic-message-routing)# cluster cluster-C
broker-X1(...uting/dynamic-message-routing/cluster)# link broker-B
broker-X1(...uting/dynamic-message-routing/cluster)# shutdown
broker-X1(...uting/dynamic-message-routing/cluster)# exit
broker-X1(...uting/dynamic-message-routing/cluster)# no link broker-B

    broker-X2(configure)# routing
broker-X2(configure/routing)# dynamic-message-routing
broker-X2(...igure/routing/dynamic-message-routing)# cluster cluster-C
broker-X2(...uting/dynamic-message-routing/cluster)# link broker-B
broker-X2(...uting/dynamic-message-routing/cluster)# shutdown
broker-X2(...uting/dynamic-message-routing/cluster)# exit
broker-X2(...uting/dynamic-message-routing/cluster)# no link broker-B

    ...

    The resulting cluster looks like the following:

  2. Disable and remove cluster C on B: 
    broker-B(configure)# routing
broker-B(configure/routing)# dynamic-message-routing
broker-B(...igure/routing/dynamic-message-routing)# cluster cluster-C
broker-B(...igure/routing/dynamic-message-routing)# shutdown
broker-B(...igure/routing/dynamic-message-routing)# exit
broker-B(...igure/routing/dynamic-message-routing)# no cluster cluster-C
  3. Disable and remove the cluster link on A to B (as in step 1).

    Node B is no longer part of the cluster, but remains the replication mate of A, as shown in the following diagram:

  4. If a topology issue One-way link between B and A is present on A, wait for it to clear. This may take several minutes.

    Proceeding to the next step before this topology issue clears may result in loss of persistent messages. Note that the presence of topology issue External link missing between B and X is normal at this step.

  5. Disable and remove replication as normal between A and B.
    1. Disable replication for all Message VPNs of both A and B: 
      broker-A(configure)# message-vpn vpn-one
broker-A(configure/message-vpn)# replication shutdown
broker-A(configure/message-vpn)# exit
broker-A(configure)# message-vpn vpn-two
broker-A(configure/message-vpn)# replication shutdown
broker-A(configure/message-vpn)# exit
...
      broker-B(configure)# message-vpn vpn-one
broker-B(configure/message-vpn)# replication shutdown
broker-B(configure/message-vpn)# exit
broker-B(configure)# message-vpn vpn-two
broker-B(configure/message-vpn)# replication shutdown
broker-B(configure/message-vpn)# exit
...
    2. Remove global replication attributes on both A and B: 
      broker-A(configure)# replication
broker-A(configure/replication)# no interface
broker-A(configure/replication)# no mate broker-B
broker-A(configure/replication)# no mate connect-via <addr-port>
      broker-B(configure)# replication
broker-B(configure/replication)# no interface
broker-B(configure/replication)# no mate broker-A
broker-B(configure/replication)# no mate connect-via <addr-port>

      In the preceding code, <addr-port> is the IP address or hostname of the event broker in the form IP-address:port or hostname:port, where the IP address is specified in the dotted decimal notation form nnn.nnn.nnn.nnn and port is the port number on the replication mate virtual router (specified as a decimal value from 0 to 65535).

  6. Verify that the cluster is UP on all brokers. See Displaying Cluster Information.

At this point B is fully removed from the DMR network and is no longer the replication mate of A, as shown below:

Pairing Two Nodes for Replication in a DMR Cluster

It is service interrupting to make two nodes already part of a DMR network into a replication group—any clients that were hosted by one broker must be rehosted on the other. During the procedure it is expected that the state of the cluster will be reported as DOWN and topology errors will be raised for a period of time as links are being adjusted between nodes. Regardless of these errors, traffic will continue to flow through the network.

On one of the brokers to be paired (in this case, B), messages that are either destined for or currently enqueued on that broker will be lost.

Assumptions

  • brokers A and B are both in cluster C, joined by an internal link
  • brokers A and B have existing DMR links (internal or external) to brokers X1 through Xn
  • the state of the clusters on all of A, B, and X1..n is UP
  • broker B is to become the replication mate of A
  • all brokers using HA or replication have Config-Sync enabled and UP (however, if B is a non-redundant broker Config-Sync is operationally DOWN on B until step 7, below)
  • clients of the replication group A/B have been configured to include the addresses of both A and B in their host lists

High Availability Considerations

  • For software event brokers, you may have to provide a second connect-via address when you configure replication and links in the steps that follow. For example, when you configure the cluster link on A to B/B', you may have to provide the addresses for both B and B'.
  • Enabling/disabling of the msg-backbone must be done on both B and B'.

Steps

  1. Shut down the msg-backbone service on B. Any clients connected to B will move to A. 
    broker-B(configure)# service msg-backbone shutdown
  2. Shut down and remove all cluster links on B to brokers X1 through Xn
    broker-B(configure)# routing
broker-B(configure/routing)# dynamic-message-routing
broker-B(...igure/routing/dynamic-message-routing)# cluster cluster-C
broker-B(...uting/dynamic-message-routing/cluster)# link broker-X1
broker-B(...uting/dynamic-message-routing/cluster)# shutdown
broker-B(...uting/dynamic-message-routing/cluster)# exit
broker-B(...uting/dynamic-message-routing/cluster)# no link broker-X1
broker-B(...uting/dynamic-message-routing/cluster)# link broker-X2
broker-B(...uting/dynamic-message-routing/cluster)# shutdown
broker-B(...uting/dynamic-message-routing/cluster)# exit
broker-B(...uting/dynamic-message-routing/cluster)# no link broker-X2
...
broker-B(...uting/dynamic-message-routing/cluster)# link broker-Xn
broker-B(...uting/dynamic-message-routing/cluster)# shutdown
broker-B(...uting/dynamic-message-routing/cluster)# exit
broker-B(...uting/dynamic-message-routing/cluster)# no link broker-Xn
  3. On each broker X1 through Xn, disable and remove the cluster link to B. 
    broker-X1(configure)# routing
broker-X1(configure/routing)# dynamic-message-routing
broker-X1(...igure/routing/dynamic-message-routing)# cluster <cluster-name>
broker-X1(...uting/dynamic-message-routing/cluster)# link broker-B
broker-X1(...uting/dynamic-message-routing/cluster)# shutdown
broker-X1(...uting/dynamic-message-routing/cluster)# exit
broker-X1(...uting/dynamic-message-routing/cluster)# no link broker-B
    broker-X2(configure)# routing
broker-X2(configure/routing)# dynamic-message-routing
broker-X2(...igure/routing/dynamic-message-routing)# cluster <cluster-name>
broker-X2(...uting/dynamic-message-routing/cluster)# link broker-B
broker-X2(...uting/dynamic-message-routing/cluster)# shutdown
broker-X2(...uting/dynamic-message-routing/cluster)# exit
broker-X2(...uting/dynamic-message-routing/cluster)# no link broker-B
    ...

    Node B is now connected only to Node A, as shown below:

  4. Configure replication as normal between A and B, such that A is active and B standby:
    1. Set global replication attributes on both A and B, such that A is the replication mate of B and vice-versa. See Configuring Replication Mates.
    2. Create all Message VPNs on B that will be replicated from A. See Configuring Message VPNs.
    3. Enable replication for Message VPNs of A (active) and B (standby). See Enabling Replication.

  5. Remove the connect-via settings of the cluster link on A to B and on B to A. 
    broker-A(configure)# routing
broker-A(configure/routing)# dynamic-message-routing
broker-A(...igure/routing/dynamic-message-routing)# cluster cluster-C
broker-A(...uting/dynamic-message-routing/cluster)# link broker-B
broker-A(.../dynamic-message-routing/cluster/link)# shutdown
broker-A(.../dynamic-message-routing/cluster/link)# no connect-via <addr-port>
…
broker-A(...uting/dynamic-message-routing/cluster)# link broker-B
broker-A(.../dynamic-message-routing/cluster/link)# no shutdown
    broker-B(configure)# routing
broker-B(configure/routing)# dynamic-message-routing
broker-B(...igure/routing/dynamic-message-routing)# cluster cluster-C
broker-B(...uting/dynamic-message-routing/cluster)# link broker-A
broker-B(.../dynamic-message-routing/cluster/link)# shutdown
broker-B(.../dynamic-message-routing/cluster/link)# no connect-via <addr-port>
…
broker-B(...uting/dynamic-message-routing/cluster)# link broker-A
broker-B(.../dynamic-message-routing/cluster/link)# no shutdown

    In the preceding code, <addr-port> is the IP address or hostname of the event broker in the form IP-address:port or hostname:port, where the IP address is specified in the dotted decimal notation form nnn.nnn.nnn.nnn and port is the port number on the replication mate virtual router (specified as a decimal value from 0 to 65535).

  6. For each broker X1 through Xn do the following:
    1. Wait for the cluster on X to report a missing link between X and B.
    2. Create a new link on B to X (see Configuring Cluster Links). At a minimum, configure these link attributes:
      • span to either internal or external as appropriate
      • authentication settings
      • connect-via
      • enabled
      If the span is external, create the corresponding DMR bridges on B (see Configuring a DMR Bridge).

  7. Enable the msg-backbone service on B.
    broker-B(configure)# no service msg-backbone shutdown
  8. Wait for Config-Sync on A and B to report all replicated Message VPNs as being in-sync.
  9. Verify that the link between A and B is UP. See Displaying Cluster Link Information.
  10. For each broker X1 through Xn do the following:

    Create a new link on X to B (see Configuring Cluster Links). At a minimum, configure these link attributes:

    • span to either internal or external as appropriate
    • authentication settings
    • connect-via
    • enabled

    If the span is external, create the corresponding DMR bridges on X (see Configuring a DMR Bridge).

  11. Verify that the links between X1 through Xn and B are UP. See Displaying Cluster Link Information.
  12. Verify that the cluster is UP on all brokers. See Displaying Cluster Information.

The new replication mate B is now fully integrated into the DMR network.

Optional: You can now move selected replicated Message VPNs on A such that they are active on B.

Adding a Replication Group to a DMR Cluster

You can add an existing replication group as new nodes to a DMR network without causing a service interruption to existing clients. During the procedure it is expected that the state of the cluster will report as being DOWN and topology errors will be raised for a period of time as links are being added between nodes. Regardless of this, existing traffic flow through the network will continue, and no message loss is expected.

Assumptions

  • brokers A and B are a replication group without a cluster configured
  • all replicated Message VPNs on A and B are active on A and standby on B
  • brokers X1 through Xn belong to cluster C
  • the state of the cluster on all of X1..n is UP
  • all brokers using HA or replication have Config-Sync enabled and UP

High Availability Considerations

For software event brokers, you may have to provide a second connect-via address when you configure replication and links in the steps that follow. For example, when you configure the cluster link on A to B/B', you may have to provide the addresses for both B and B'.

Steps

  1. Create cluster C on brokers A and B, and configure its authentication and other attributes as needed. See Configuring a DMR Cluster.

  2. On A, enable DMR for every Message VPN on X1 through Xn for which DMR is enabled.
    broker-A(configure)# message-vpn vpn-one
broker-A(configure/message-vpn)# dynamic-message-routing
broker-A(...e/message-vpn/dynamic-message-routing)# no shutdown
broker-A(configure/message-vpn)# exit
broker-A(configure)# message-vpn vpn-two
broker-A(configure/message-vpn)# dynamic-message-routing
broker-A(...e/message-vpn/dynamic-message-routing)# no shutdown
broker-A(configure/message-vpn)# exit
...
  3. Create new cluster links from A to B and from B to A (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via must NOT be set
    • enabled

  4. Verify that the link between A and B is UP (see Displaying Cluster Link Information), and that cluster C is UP on both A and B (see Displaying Cluster Information.).
  5. Create a new cluster link on A to each broker X1 through Xn (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via
    • enabled
  6. On each broker X1 through Xn, create a new link to A (see Configuring Cluster Links). At a minimum, configure these link attributes: :
    • span to be internal
    • authentication settings
    • connect-via
    • enabled
  7. Verify that the links between X1 through Xn and A are UP (see Displaying Cluster Link Information).

  8. Configure a new link on B to each broker X1 through Xn (see Configuring Cluster Links). At a minimum, configure these link attributes:
    • span to be internal
    • authentication settings
    • connect-via
    • enabled
  9. For each broker X1 through Xn do the following:
    1. Wait for the cluster on X to report a topology issue which can be either:
      • a missing link between X and B
      • a one-way link between X and B
    2. Create a new link on X to B (see Configuring Cluster Links). At a minimum, configure these link attributes:
      • span to be internal
      • authentication settings
      • connect-via
      • enabled
  10. Verify that the cluster is UP on all brokers. See Displaying Cluster Information.

At this point, the new replication group A/B is fully integrated into the DMR network. If required, you can move the selected replicated Message VPNs on A so that they are active on B.