HA Group Configuration

In this section you'll learn how to set up a high-availability (HA) redundancy group of Solace PubSub+ software event brokers.

The steps shown in this topic are generally applicable to HA group configuration, but if you need to set up an HA group on a Windows or macOS platform, instructions tailored for doing just that can be found on these pages:

• HA Group Configuration for Windows
• HA Group Configuration for macOS

If you're interested in setting up an HA group in AWS, take a look at the following Quick Start in GitHub that will take you through the steps of HA configuration:

• Install and Configure Solace PubSub+ Software Event Brokers in an HA Tuple using AWS Cloud Formation

Although the example shown in this topic uses Solace CLI, the intent is to help you become familiar with HA configuration as a step towards using more advanced, production-oriented techniques. For instance, you can also get an HA group up-and-running using configuration keys, as described in Configuring HA Groups Using Configuration Keys.

Step 1: Review Prerequisites & Configuration Parameters

Before configuring an HA redundancy group, you need to review the following:

• Prerequisites
• Configuration Parameters
• Considerations When Using Bridge Networking

Prerequisites

The configuration steps provided in this section assume that:

• You have obtained a Solace PubSub+ software event broker package. To download a copy of a software event broker package, visit Solace PubSub+ software event broker downloads page.
• The software event brokers to be used in the HA redundancy group use:
  • Configuration defaults.
  • Host networking and single tenant hosts.

With regard to the above assumptions, the only configuration changes required are the following:

1. Each software event broker in the HA redundancy group is configured with a unique host and router name (see Hostname Configuration).
2. Each software event broker in the HA redundancy group is configured with a unique IP address associated with the Management VRF.
3. All software event brokers in the HA redundancy group must have the same release version.
4. All messaging nodes within the HA group configured to use the same maximum number of client connections. When upgrading a monitoring node of an HA group to Solace PubSub+ version 8.10 or higher, the database is transformed to use 100 as the maximum number of client connections (see System Resource Requirements).
5. System clocks in both the primary and backup event brokers are synchronized with a Network Time Protocol (NTP) server (see the Software Event Brokers section of Synchronizing Clocks with NTP Servers).
6. Each software event broker in the HA redundancy group is deployed on a different physical host system.
7. The primary and backup software event brokers are run in identical virtual machines (with the same amount of virtual RAM and same number of virtual processor cores) and use the same size disks for their message spools.
8. If you are deploying the HA redundancy group in a cloud environment, you must ensure that security rules are created for ports 8300, 8301, 8302 (the three HA configuration synchronization ports), 8741 (the HA mate link port), and 55555 (the SMF port). See the firewall rule tables, shown below, for details.
9. In the example discussed in this section, the default HA mate link port is used. If you need to use some other port in your deployment, you'll need to use the configure/redundancy/mate-link/port <port> Solace CLI command to set the value. For more information about the command, you can look it up in the Software Event Broker CLI Commands.

   Inbound Firewall Rules for Solace PubSub+ software event broker 1

   Permission	Protocol	Source	Destination	Port
   Allow	TCP	Solace PubSub+ software event broker 2 IP Monitoring Node IP	Solace PubSub+ software event broker 1 IP	8300
   Allow	TCP and UDP	Solace PubSub+ software event broker 2 IP Monitoring Node IP	Solace PubSub+ software event broker 1 IP	8301
   Allow	TCP and UDP	Solace PubSub+ software event broker 2 IP Monitoring Node IP	Solace PubSub+ software event broker 1 IP	8302
   Allow	TCP	Solace PubSub+ software event broker 2 IP	Solace PubSub+ software event broker 1 IP	8741
   Allow	TCP	Solace PubSub+ software event broker 2 IP	Solace PubSub+ software event broker 1 IP	55555

   Inbound Firewall Rules for Solace PubSub+ software event broker 2

   Permission	Protocol	Source	Destination	Port
   Allow	TCP	Solace PubSub+ software event broker 1 IP Monitoring Node IP	Solace PubSub+ software event broker 2 IP	8300
   Allow	TCP and UDP	Solace PubSub+ software event broker 1 IP Monitoring Node IP	Solace PubSub+ software event broker 2 IP	8301
   Allow	TCP and UDP	Solace PubSub+ software event broker 1 IP Monitoring Node IP	Solace PubSub+ software event broker 2 IP	8302
   Allow	TCP	Solace PubSub+ software event broker 1 IP	Solace PubSub+ software event broker 2 IP	8741
   Allow	TCP	Solace PubSub+ software event broker 1 IP	Solace PubSub+ software event broker 2 IP	55555

   Inbound Firewall Rules for Monitoring Node

   Permission	Protocol	Source	Destination	Port
   Allow	TCP	Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP	Monitoring Node IP	8300
   Allow	TCP and UDP	Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP	Monitoring Node IP	8301
   Allow	TCP and UDP	Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP	Monitoring Node IP	8302

Configuration Parameters

The following table lists parameter values that are required in the configuration procedure.

Parameter	Description	Example Values
Interface Name	The name of the interface the Management VRF uses. To determine the name, enter the show ip vrf management command—it is the name displayed in the Interface column without the “:<#>” suffix.	intf0
IP Interface Name	The IP interface name is the interface name concatenated with a “:1”.	intf0:1
Pre-Shared Authentication Key	32 to 256 bytes of binary data encoded in base 64. Must be the same for all members of the HA group.	MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDEK
IP Address	A Solace PubSub+ software event broker's unique IP address/netmask. To determine the IP address, enter the show ip vrf management command—it is the IP address displayed in the output for the Static V Router.	Primary software event broker: 192.168.132.16/20 Backup software event broker: 192.168.132.39/20 Monitoring node: 192.168.132.3/20
HA redundancy group parameters	The Solace PubSub+ software event broker name, IP address or fully qualified domain name (FQDN), and the port number for each node. The node name must be the same as the router name. Redundancy requires three ports, which must be accessible by all nodes in a redundancy group. Port 8300 is set by default, and the two subsequent ports are used as required.	Primary event broker: solace1, connect-via 192.168.132.16 Backup event broker: solace2, connect-via 192.168.132.39 Monitoring node: solace3, connect-via 192.168.132.3
	A node role assigned for each node.	Primary: active-standby-role primary Backup: active-standby-role backup Monitoring: Monitoring node doesn't have the active-standby-role assigned.

Considerations When Using Bridge Networking

If you configure a Solace PubSub+ software event broker container to use bridge networking, the three redundancy ports (which defaults to 8300, 8301, 8302) must be published. You should ensure the following ports are accessible:

• port 8300 must be published for TCP
• ports 8301 and 8302 must be published for TCP and UDP

For example, docker create API supports the -p flag or --publish flag to publish ports. A typical docker create command line would include: -p 8300:8300 -p 8301:8301 -p 8302:8302 -p 8301:8301/udp -p 8302:8302/udp.

For more information on the use of Bridge Networking in the configuration of HA groups, refer to Container Networking.

Step 2: Configure the Nodes

To configure an HA redundancy group, you must configure the primary, backup, and monitoring nodes for Direct messaging. Once all the three nodes are configured, they are effectively enabled for Direct Messaging.

To configure the Solace PubSub+ software event broker redundancy nodes, perform the following steps in order:

1. Configure the Primary Event Broker
2. Configure the Backup Event Broker
3. Configure the Monitoring Node

Before configuring Solace PubSub+ software event broker redundancy, we recommend that you first shut down the message-spool.

Configure the Primary Event Broker

To configure the primary event broker for an HA redundancy group, perform the following steps:

1. On the event broker that will act as the primary (solace1 in the examples), shut down the message spool:

   solace1> enable
   solace1# configure
   solace1(configure)# hardware message-spool shutdown
   All message spooling will be stopped.
   Do you want to continue (y/n)? y

2. Configure the hostlist failover mechanism:

   solace1(configure)# redundancy
   solace1(configure/redundancy)# switchover-mechanism hostlist
   solace1(configure/redundancy)# exit
   

3. Configure the HA redundancy group, assign an active-standby-role of primary to this event broker, and then enable redundancy:

   solace1(configure)# redundancy
   solace1(configure/redundancy)# group
   solace1(configure/redundancy/group)# create node solace1
   solace1(configure/redundancy/group/node)# connect-via 192.168.132.16
   solace1(configure/redundancy/group/node)# node-type message-routing-node
   solace1(configure/redundancy/group/node)# exit
   solace1(configure/redundancy/group)# create node solace2
   solace1(configure/redundancy/group/node)# connect-via 192.168.132.39
   solace1(configure/redundancy/group/node)# node-type message-routing-node
   solace1(configure/redundancy/group/node)# exit
   solace1(configure/redundancy/group)# create node solace3
   solace1(configure/redundancy/group/node)# connect-via 192.168.132.3
   solace1(configure/redundancy/group/node)# node-type monitor-node
   solace1(configure/redundancy/group/node)# exit
   solace1(configure/redundancy/group)# exit
   solace1(configure/redundancy)# authentication
   solace1(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
   solace1(configure/redundancy/authentication)# exit
   solace1(configure/redundancy)# active-standby-role primary
   solace1(configure/redundancy)# no shutdown

   Where:

   <pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

   In release 9.1.0 and earlier, you must use the redundancy group password <password> command to configure the group password for authentication between the nodes. In these releases, group passwords must be between 1 and 128 characters and are required to be the same on each node.

Configure the Backup Event Broker

To configure the backup event broker for an HA redundancy group, perform the following steps:

1. On the event broker that will act as the backup (solace2 in the examples), shut down the message spool:

solace2> enable
solace2# configure
solace2(configure)# hardware message-spool shutdown
All message spooling will be stopped
Do you want to continue (y/n)? y

2. Configure hostlist as the failover mechanism:

   solace2(configure)# redundancy
   solace2(configure/redundancy)# switchover-mechanism hostlist

3. Configure the HA redundancy group, assign an active-standby-role of backup to this event broker, and then enable redundancy:

   solace2(configure)# redundancy
   solace2(configure/redundancy)#  group
   solace2(configure/redundancy/group)# create node solace1
   solace2(configure/redundancy/group/node)# connect-via 192.168.132.16
   solace2(configure/redundancy/group/node)# node-type message-routing-node
   solace2(configure/redundancy/group/node)# exit
   solace2(configure/redundancy/group)# create node solace2
   solace2(configure/redundancy/group/node)# connect-via 192.168.132.39
   solace2(configure/redundancy/group/node)# node-type message-routing-node
   solace2(configure/redundancy/group/node)# exit
   solace2(configure/redundancy/group)# create node solace3
   solace2(configure/redundancy/group/node)# connect-via 192.168.132.3
   solace2(configure/redundancy/group/node)# node-type monitor-node
   solace2(configure/redundancy/group/node)# exit
   solace2(configure/redundancy/group)# exit
   solace2(configure/redundancy)# authentication
   solace2(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
   solace2(configure/redundancy/authentication)# exit
   solace2(configure/redundancy)# active-standby-role backup
   solace2(configure/redundancy)# no shutdown

   Where:

   <pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

   In release 9.1.0 and earlier, you must use the redundancy group password <password> command to configure the group password for authentication between the nodes. In these releases, group passwords must be between 1 and 128 characters and are required to be the same on each node.

Configure the Monitoring Node

To configure the monitoring node for an HA redundancy group, perform the following steps:

1. On the event broker that will act as the monitoring node (solace3 in the examples), reload the default configuration:

   solace3# reload default-config monitoring-node
   This command causes a reload of the system
   Do you want to continue (y/n)? y

   For Solace PubSub+ software event brokers running as containers, after reloading the default configuration for the monitoring node, you must start the container manually.

2. Configure the hostlist failover mechanism:

   solace3(configure)# redundancy
   solace3(configure/redundancy)# switchover-mechanism hostlist

3. Configure the HA redundancy group and enable redundancy.

   Notice that as a monitoring node, this event broker doesn't have the active-standby-role assigned.

   solace3(configure)# redundancy
   solace3(configure/redundancy)# group
   solace3(configure/redundancy/group)# create node solace1
   solace3(configure/redundancy/group/node)# connect-via 192.168.132.16
   solace3(configure/redundancy/group/node)# node-type message-routing-node
   solace3(configure/redundancy/group/node)# exit
   solace3(configure/redundancy/group)# create node solace2
   solace3(configure/redundancy/group/node)# connect-via 192.168.132.39
   solace3(configure/redundancy/group/node)# node-type message-routing-node
   solace3(configure/redundancy/group/node)# exit
   solace3(configure/redundancy/group)# create node solace3
   solace3(configure/redundancy/group/node)# connect-via 192.168.132.3
   solace3(configure/redundancy/group/node)# node-type monitor-node
   solace3(configure/redundancy/group/node)# exit
   solace3(configure/redundancy/group)# exit
   solace3(configure/redundancy)# authentication
   solace3(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
   solace3(configure/redundancy/authentication)# exit
   solace3(configure/redundancy)# no shutdown

   Where:

   <pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

   In release 9.1.0 and earlier, you must use the redundancy group password <password> command to configure the group password for authentication between the nodes. In these releases, group passwords must be between 1 and 128 characters and are required to be the same on each node.

Step 3: Enable Guaranteed Messaging

By default, an HA group has Guaranteed messaging disabled. It can only be enabled after the primary, backup, and monitoring nodes have redundancy configured, and although the use of Guaranteed messaging is optional, it's recommended because it's required for the group to:

• Accept clients that send or receive Guaranteed messages.
• Use Config-Sync.

Prior to release 9.2.0, mate-link connect-via [<addr-port>] was used to start the mate-link connection. In release 9.2.0+, mate-link connect via command has been deprecated and instead the connect-via address from the group node is used by default. In the example below, the default HA mate link port is used, if you need to use some other port in your deployment, execute the configure/redundancy/mate-link/port <port> Solace CLI command to set the value.

To enable Guaranteed messaging for an HA group, perform the following steps on the primary and backup event brokers:

1. Enable the message spool on both event brokers.

   On the primary, enter the following commands:

   solace1(configure)# hardware message-spool
   solace1(configure/hardware/message-spool)# no shutdown
   solace1(configure/hardware/message-spool)# exit
   solace1(configure/hardware)# exit
   

   On the backup, enter the following commands:

   solace2(configure)# hardware message-spool
   solace2(configure/hardware/message-spool)# no shutdown
   solace2(configure/hardware/message-spool)# exit
   solace2(configure/hardware)# exit

2. Start the mate-link service on both event brokers.

   On the primary, enter the following commands:

   solace1(configure)# service mate-link
   solace1(configure/service/mate-link)# no shutdown
   solace1(configure/service/mate-link)# exit
   solace1(configure/service)# exit

   On the backup, enter the following commands:

   solace2(configure)# service mate-link
   solace2(configure/service/mate-link)# no shutdown
   solace2(configure/service/mate-link)# exit
   solace2(configure/service)# exit

3. Optional: You can enable encryption on the mate-link.

On the primary, enter the following commands:

solace1(configure)# redundancy mate-link
solace1(configure/redundancy/mate-link)# ssl
solace1(configure/redundancy/mate-link)# exit
solace1(configure/redundancy)# exit

On the backup, enter the following commands:

solace2(configure)# redundancy mate-link
solace2(configure/redundancy/mate-link)# ssl
solace2(configure/redundancy/mate-link)# exit
solace2(configure/redundancy)# exit

4. To verify that Guaranteed messaging for the group is enabled, use the show redundancy command. The values of ADB Link To Mate, ADB Hello To Mate, and Message Spool Status are what you need to look at.

   On the primary you should see Up and AD-Active:

   solace1(configure/redundancy)# show redundancy
   Configuration Status     : Enabled
       
   . . . 
       
   ADB Link To Mate         : Up
   ADB Hello To Mate        : Up
       
   . . . 
   
   Message Spool Status           AD-Active

   Likewise, on the backup you should see Up and AD-Standby:

   solace2(configure/redundancy)# show redundancy
   Configuration Status     : Enabled
   
   . . . 
   
   ADB Link To Mate         : Up
   ADB Hello To Mate        : Up
   
   . . . 
   
   Message Spool Status           AD-Standby

Step 4: Enable Config-Sync

After a Solace PubSub+ software event broker HA redundancy group is configured to support Guaranteed messaging, use Config-Sync to synchronize configurations between primary and backup event brokers.

For Config-Sync to be enabled, make sure that the inbound TCP rule at port 55555 is allowed for the mate event broker to connect. See Prerequisites for more information.

Set up Config-Sync without TLS

To enable Config-Sync for the HA group, perform the following steps:

1. Enable Config-Sync.

   On the primary (solace1) event broker enter the following commands:

   solace1(configure)# config-sync
   solace1(configure/config-sync)# no shutdown
   solace1(configure/config-sync)# exit
   solace1(configure)# exit
   

   On the backup (solace2) event broker enter the following commands:

   solace2(configure)# config-sync
   solace2(configure/config-sync)# no shutdown
   solace2(configure/config-sync)# exit
   solace2(configure)# exit

2. Assert the primary event broker's configuration.

   When enabling Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the leader event broker (usually it's the primary event broker in the pair) over its mate.

   As a result of this operation the Oper Status of Config-Sync should change from Down to Up on both primary and backup event brokers (check it with the show config-sync User EXEC command).

   solace1# admin
   solace1(admin)# config-sync
   solace1(admin/config-sync)# assert-leader router
   Processed 1 config-sync tables.

3. Assert one Message VPN's configuration.

   If the Message VPNs' configurations are out-of-sync, you must manually synchronize a Message VPN configuration between two event brokers. To do this, you must select one of the message VPNs as a leader (it doesn't have to be on the leader or primary event broker), and assert its configuration over the same Message VPN on the other event broker.

   solace1(admin/config-sync)# assert-leader message-vpn myvpn
   WARNING: This command can temporarily disconnect clients on the AD-inactive appliance. As well in-flight messages may not be delivered
   to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance.
   Do you want to continue (y/n)? y
   Processed 1 config-sync tables.

Set up Config-Sync over TLS

To enable Config-Sync over TLS for the HA group, perform the following steps.

All event brokers in the HA group must be configured with the same pre-shared authentication key as shown in the previous steps to be able to enable Config-Sync over TLS.

1. Enable Config-Sync.

   On the primary (solace1) event broker enter the following commands:

   solace1(configure)# config-sync
   solace1(configure/config-sync)# ssl
   solace1(configure/config-sync)# no shutdown
   solace1(configure/config-sync)# exit
   solace1(configure)# exit
   

   On the backup (solace2) event broker enter the following commands:

   solace2(configure)# config-sync
   solace2(configure/config-sync)# ssl
   solace2(configure/config-sync)# no shutdown
   solace2(configure/config-sync)# exit
   solace2(configure)# exit

2. Assert the primary event broker's configuration.

   When enabling Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the leader event broker (usually it's the primary event broker in the pair) over its mate.

   As a result of this operation the Oper Status of Config-Sync should change from Down to Up on both primary and backup event brokers (check it with the show config-sync User EXEC command).

   solace1# admin
   solace1(admin)# config-sync
   solace1(admin/config-sync)# assert-leader router
   Processed 1 config-sync tables.

3. Assert one Message VPN's configuration.

   If the Message VPNs' configurations are out-of-sync, you must manually synchronize a Message VPN configuration between two event brokers. To do this, you must select one of the message VPNs as a leader (it doesn't have to be on the leader or primary event broker), and assert its configuration over the same Message VPN on the other event broker.

   solace1(admin/config-sync)# assert-leader message-vpn myvpn
   WARNING: This command can temporarily disconnect clients on the AD-inactive appliance. As well in-flight messages may not be delivered
   to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance.
   Do you want to continue (y/n)? y
   Processed 1 config-sync tables.

Step 5: Validate Failover

To validate the HA group's failover operation, perform the following steps:

1. Manually release activity on the primary event broker (see Releasing Event Broker Activity), and test that messages are being published and received by the backup event broker.

   solace1> enable
   solace1# configure
   solace1(configure)# redundancy
   solace1(configure/redundancy)# release-activity

   Note that after running the release-activity command, the backup node will be up, but the redundancy will be down. You must run the no release-activity, as shown in Step 3 below, to ensure the primary node rejoins the HA group, and the redundancy is back up.

2. Validate that all the clients successfully reconnect to the backup event broker after activity is released on the primary event broker, and test messages continue to be published and received as expected.

   Use the show stats client command to check the client connections on the backup event broker. The total number of connected clients on the back event broker must same as it was in the primary event broker.

   Example :

   solace2# show stats client
   Total Clients:            4
   Total Clients Connected:  4
   . . .
   

3. Manually take the activity back to the primary event broker.

   solace1(configure/redundancy)# no release-activity
   solace1(configure/redundancy)# home

4. Force the backup event broker to give up activity (see Forcing Backups to Give Up Activity to Primaries).

   solace2> enable
   solace2# admin
   solace2(admin)# redundancy
   solace2(admin/redundancy)# revert-activity

5. Validate that all the clients reconnect back to the primary event broker after the activity is released on the backup event broker, and test that messages continue to be published and received as expected.

   Use the show stats client command to check the client connections on the primary event broker.

   Example :

   solace1# show stats client
   Total Clients:            4
   Total Clients Connected:  4
   . . .
   

Next Steps

You now have three Solace PubSub+ software event brokers configured in an HA group. You can now do things like use the SDKPerf tool to test messaging, configure an event broker's health check related settings, or configure the maximum number of client connections.

• Download SDKPerf—To get started, see SDKPerf's Quick Start guide.
• Load Balancer Health Checks—If the Solace PubSub+ software event brokers are operating in conjunction with an environment provided load balancer, it is recommended to review the Load Balancer Health Checks feature page.
• Scaling Up an Existing HA Group—Configure system scaling parameters for HA redundancy groups using the Solace CLI.