Horizontal Scaling Configuration

The following sections provides an example of how you can use Dynamic Message Routing (DMR) to scale the capabilities of an individual event broker.

This example shows you how to create a single cluster with three nodes (where each node can consist of either a single event broker or a High Availability (HA) group). Together, the three nodes increase the connection limits, subscription limits, queue limits, and bandwidth beyond a single node, without requiring additional subscription management.

Before you can enable DMR horizontal scaling, you must ensure that the requirements discussed in the Prerequisites section are met.

Using This Example

In this example for this configuration, a cluster contains three event brokers (nodes) with internal links configured to connect the nodes.

The example below show a cluster with nodes that have placeholder names using the notation Broker-<letter>, which represents the router name for that node (where each may be a standalone broker or the primary broker in an HA group). Keep in mind that in the sample commands below, the IP addresses, names of the Message VPN, and the names Cluster-A, Broker-A, Broker-B, and Broker-C are placeholders that you should replace with your own values.

Step 1: Create a Secured Cluster

Before you can link your network of nodes together, you must create a cluster to contain the nodes.

In this example, a single cluster is created for the three participating nodes, and basic authentication along with a shared password (myshared_secret123) is used to secure the cluster.

The names Broker-A, Broker-B, and Broker-C are the placeholder router names of the event brokers we are linking (where each may be a standalone broker or the primary broker in an HA group) and Cluster-A is the name of the cluster.

To create a single secured cluster for all three nodes:

Broker-A> enable
Broker-A# config
Broker-A(configure)# routing
Broker-A(configure/routing)# dynamic-message-routing
Broker-A(...igure/routing/dynamic-message-routing)# create cluster Cluster-A
Broker-A(...uting/dynamic-message-routing/cluster)# authentication
Broker-A(...essage-routing/cluster/authentication)# basic
Broker-A(...-routing/cluster/authentication/basic)# password myshared_secret123
Broker-A(...-routing/cluster/authentication/basic)# no shutdown
Broker-A(...-routing/cluster/authentication/basic)# exit
Broker-A(...essage-routing/cluster/authentication)# exit
Broker-A(...uting/dynamic-message-routing/cluster)# no shutdown

To add Broker-B to the cluster, log in to Broker-B and create a secured cluster with the same name and password as the cluster on Broker-A:

Broker-B> enable
Broker-B# config
Broker-B(configure)# routing
Broker-B(configure/routing)# dynamic-message-routing
Broker-B(...igure/routing/dynamic-message-routing)# create cluster Cluster-A
Broker-B(...uting/dynamic-message-routing/cluster)# authentication
Broker-B(...essage-routing/cluster/authentication)# basic
Broker-B(...-routing/cluster/authentication/basic)# password myshared_secret123
Broker-B(...-routing/cluster/authentication/basic)# no shutdown
Broker-B(...-routing/cluster/authentication/basic)# exit
Broker-B(...essage-routing/cluster/authentication)# exit
Broker-B(...uting/dynamic-message-routing/cluster)# no shutdown

To add Broker-C to the cluster, log in to Broker-C and create a secured cluster with the same name and password as the cluster on Broker-A:

Broker-C> enable
Broker-C# config
Broker-C(configure)# routing
Broker-C(configure/routing)# dynamic-message-routing
Broker-C(...igure/routing/dynamic-message-routing)# create cluster Cluster-A
Broker-C(...uting/dynamic-message-routing/cluster)# authentication
Broker-C(...essage-routing/cluster/authentication)# basic
Broker-C(...-routing/cluster/authentication/basic)# password myshared_secret123
Broker-C(...-routing/cluster/authentication/basic)# no shutdown
Broker-C(...-routing/cluster/authentication/basic)# exit
Broker-C(...essage-routing/cluster/authentication)# exit
Broker-C(...uting/dynamic-message-routing/cluster)# no shutdown

Step 2: Create Cluster Links for Each Node

After you create a cluster, you can create a bidirectional link between nodes to establish a control channel. Once a control channel is established, the subscription sets needed by each node can be exchanged with the other. In addition, the subscription sets are dynamically kept up-to-date as local client and queue subscriptions change.

In this example, a full mesh of cluster links is configured between the three nodes, which is a requirement for horizontal scaling (that is, using internal links). This configuration creates a control channel across the three nodes.

For production deployments, we recommend using the local and remote link initiator configurations.

To create a mesh of cluster links for the three nodes.

Broker-A> enable
Broker-A# config 
Broker-A(configure)# routing dynamic-message-routing
Broker-A(...igure/routing/dynamic-message-routing)# cluster Cluster-A
Broker-A(...uting/dynamic-message-routing/cluster)# create link Broker-B
Broker-A(.../dynamic-message-routing/cluster/link)# span internal
Broker-A(.../dynamic-message-routing/cluster/link)# initiator local
Broker-A(.../dynamic-message-routing/cluster/link)# connect-via 192.0.2.2
Broker-A(.../dynamic-message-routing/cluster/link)# transport ssl
Broker-A(.../dynamic-message-routing/cluster/link)# no shutdown
Broker-A(.../dynamic-message-routing/cluster/link)# exit
Broker-A(...uting/dynamic-message-routing/cluster)# create link Broker-C
Broker-A(.../dynamic-message-routing/cluster/link)# span internal
Broker-A(.../dynamic-message-routing/cluster/link)# initiator remote
Broker-A(.../dynamic-message-routing/cluster/link)# transport ssl
Broker-A(.../dynamic-message-routing/cluster/link)# no shutdown

Log into Broker-B and create links from Broker-B to Broker-A and Broker-C:

Broker-B> enable
Broker-B# config 
Broker-B(configure)# routing dynamic-message-routing
Broker-B(...igure/routing/dynamic-message-routing)# cluster Cluster-A
Broker-B(...uting/dynamic-message-routing/cluster)# create link Broker-A
Broker-B(.../dynamic-message-routing/cluster/link)# span internal
Broker-B(.../dynamic-message-routing/cluster/link)# initiator remote
Broker-B(.../dynamic-message-routing/cluster/link)# transport ssl
Broker-B(.../dynamic-message-routing/cluster/link)# no shutdown
Broker-B(.../dynamic-message-routing/cluster/link)# exit
Broker-B(...uting/dynamic-message-routing/cluster)# create link Broker-C
Broker-B(.../dynamic-message-routing/cluster/link)# span internal
Broker-B(.../dynamic-message-routing/cluster/link)# initiator remote
Broker-B(.../dynamic-message-routing/cluster/link)# transport ssl
Broker-B(.../dynamic-message-routing/cluster/link)# no shutdown

Broker-C> enable
Broker-C# config 
Broker-C(configure)# routing dynamic-message-routing
Broker-C(...igure/routing/dynamic-message-routing)# cluster Cluster-A
Broker-C(...uting/dynamic-message-routing/cluster)# create link Broker-A
Broker-C(.../dynamic-message-routing/cluster/link)# span internal
Broker-C(.../dynamic-message-routing/cluster/link)# initiator local
Broker-C(.../dynamic-message-routing/cluster/link)# connect-via 192.0.2.1
Broker-C(.../dynamic-message-routing/cluster/link)# transport ssl
Broker-C(.../dynamic-message-routing/cluster/link)# no shutdown
Broker-C(.../dynamic-message-routing/cluster/link)# exit
Broker-C(...uting/dynamic-message-routing/cluster)# create link Broker-B
Broker-C(.../dynamic-message-routing/cluster/link)# span internal
Broker-C(.../dynamic-message-routing/cluster/link)# initiator local
Broker-C(.../dynamic-message-routing/cluster/link)# connect-via 192.0.2.2
Broker-C(.../dynamic-message-routing/cluster/link)# transport ssl
Broker-C(.../dynamic-message-routing/cluster/link)# no shutdown

Step 3: Verify All Control Channels Are Operational

Now that you've configured cluster links to support control channels between nodes, you can check the configuration by verifying that each link is operational.

To check the configuration, enter the show cluster <cluster-name-pattern> link * command on each participating node, and verify that both the Admin and Operational states are Up for all links. For example, for Cluster-A:

Broker-A> show cluster Cluster-A link *

Cluster Name      : Cluster-A
Node Name         : Broker-A

Link                               Admin Oper  Fail Reason / Uptime
---------------------------------- ----- ----- ----------------------------
#ACTIVE                            Up    Up    0d 0h 3m 35s
Broker-B                           Up    Up    0d 0h 2m 43s
Broker-C                           Up    Up    0d 0h 2m 24s

Troubleshooting Failures

If any links have an Operational state of Down, use the following show commands to gather more information:

show cluster <cluster-name-pattern> detail
show cluster <cluster-name-pattern> link <link-name-pattern> detail
show cluster <cluster-name-pattern> link <link-name-pattern> channel detail

To obtain complete information, you may need to issue these commands on both ends of the link.

Step 4: Enable DMR for Each Participating Message VPN

After you've configured the necessary cluster links, you can enable DMR for each Message VPN you want to participate in the clustering topology. This configuration establishes data channels between nodes. After a data channel has been established, application messages are exchanged across nodes, as dictated by the subscription sets maintained over control channels.

In this example, DMR is enabled for the Message VPN named red on each node, creating a data channel across the three nodes.

The names of the Message VPNs in the DMR must be the same for horizontal scaling. In this case, the Message VPN is named red. The name red is a placeholder for the Message VPN in the sample commands that you should replace with the name of your own Message VPN. See Configuring Message VPNs for more information.

To enable DMR for each participating message VPN:

Log in to Broker-A and enable DMR on Message VPN you want to participate in the clustering topology—in this case the red Message VPN.

Broker-A> enable
Broker-A# config 
Broker-A(configure)# message-vpn red
Broker-A(configure/message-vpn)# dynamic-message-routing
Broker-A(...e/message-vpn/dynamic-message-routing)# no shutdown

Broker-B> enable
Broker-B# config 
Broker-B(configure)# message-vpn red
Broker-B(configure/message-vpn)# dynamic-message-routing
Broker-B(...e/message-vpn/dynamic-message-routing)# no shutdown

Broker-C> enable
Broker-C# config 
Broker-C(configure)# message-vpn red
Broker-C(configure/message-vpn)# dynamic-message-routing
Broker-C(...e/message-vpn/dynamic-message-routing)# no shutdown

Step 5: Verify Data Channels Are Operational

At this point, you've configured all the necessary components to enable DMR on your network of PubSub+ event brokers. To verify that the DMR configuration is successful and working as intended, publish messages to a topic on one node, and ensure that you can consume the messages from a queue with a matching topic subscription on another node.

Before you start this step, you must create a queue on Broker-B with a matching subscription of a/b. For information about adding a topic, see Adding Topic Subscriptions to Queues

To verify that the data channels are operational using SDKPerf:

Publish messages to one node in the cluster.

sdkperf_c -cip=192.0.2.1:55443 -ptl=a/b -mn=50 -mr=50 -md
...
Waiting for publishing to complete.  Ctrl-c to exit.

Allow clients to finish receiving. Waiting: 2 secs
-----------------------------------------------------
Aggregate Msgs Sent Stats (Total # clients: 1)
Total Messages transmitted                 = 50
Computed publish rate (msg/sec)            = 49.99

-----------------------------------------------------
Aggregate Msgs Recv Stats (Total # clients: 1)
Total Messages received across all clients = 0
Computed receive rate (msg/sec aggregate)  = 0.00
Messages received with discard indication  = 0

Subscribe to a queue with a matching topic subscription on another node in the cluster and verify that the published traffic is received by the subscriber.

sdkperf_c -cip=192.0.2.2:55443 -sql=Q1
...
Receiving messages.  Ctrl-c to exit.

-----------------------------------------------------
Aggregate Msgs Sent Stats (Total # clients: 1)
Total Messages transmitted                 = 0
Computed publish rate (msg/sec)            = 0.00

-----------------------------------------------------
Aggregate Msgs Recv Stats (Total # clients: 1)
Total Messages received across all clients = 50
Computed receive rate (msg/sec aggregate)  = 14626.11
Messages received with discard indication  = 0

Provide feedback