HA Group Configuration
In this section you'll learn how to set up a high-availability (HA) redundancy group of Solace PubSub+ software event brokers.
The steps shown in this topic are generally applicable to HA group configuration, but if you need to set up an HA group on a Windows or macOS platform, instructions tailored for doing just that can be found on these pages:
If you're interested in setting up an HA group in AWS, take a look at the following Quick Start in GitHub that will take you through the steps of HA configuration:
- Install and Configure Solace PubSub+ Software Event Brokers in an HA Tuple using AWS Cloud Formation
Although the example shown in this topic uses Solace CLI, the intent is to help you become familiar with HA configuration as a step towards using more advanced, production-oriented techniques. For instance, you can also get an HA group up-and-running using configuration keys, as described in Configuring HA Groups Using Configuration Keys.
Step 1: Review Prerequisites & Configuration Parameters
Before configuring an HA redundancy group, you need to review the following:
Prerequisites
The configuration steps provided in this section assume that:
- You have obtained a Solace PubSub+ software event broker package. To download a copy of a software event broker package, visit Solace PubSub+ software event broker downloads page.
- The software event brokers to be used in the HA redundancy group use:
- Configuration defaults.
- Host networking and single tenant hosts.
With regard to the above assumptions, the only configuration changes required are the following:
- Each software event broker in the HA redundancy group is configured with a unique host and router name (see Hostname Configuration).
- Each software event broker in the HA redundancy group is configured with a unique IP address associated with the Management VRF (see Software Event Broker Host Interface Configuration).
- All software event brokers in the HA redundancy group must have the same release version.
- All messaging nodes within the HA group configured to use the same maximum number of client connections. When upgrading a monitoring node of an HA group to Solace PubSub+ version 8.10 or higher, the database is transformed to use 100 as the maximum number of client connections (see System Resource Requirements).
- System clocks in both the primary and backup event brokers are synchronized. with a Network Time Protocol (NTP) server (see the Software Event Brokers section of Synchronizing Clocks with NTP Servers).
- Each software event broker in the HA redundancy group is deployed on a different physical host system.
- The primary and backup software event brokers are run in identical virtual machines (with the same amount of virtual RAM and same number of virtual processor cores) and use the same size disks for their message spools.
- If you are deploying the HA redundancy group in a cloud environment, you must ensure that security rules are created for ports 8300, 8301, 8302 (the three HA configuration synchronization ports), 8741 (the HA mate link port), and 55555 (the SMF port). See the firewall rule tables, shown below, for details.
- In the example discussed in this section, the default HA mate link port is used. If you need to use some other port in your deployment, you'll need to use the
configure/redundancy/mate-link/port <port>
Solace CLI command to set the value. For more information about the command, you can look it up in the Software Event Broker CLI Commands.Inbound Firewall Rules for Solace PubSub+ software event broker 1
Permission Protocol Source Destination Port Allow TCP Solace PubSub+ software event broker 2 IP Monitoring Node IP Solace PubSub+ software event broker 1 IP 8300 Allow TCP and UDP Solace PubSub+ software event broker 2 IP Monitoring Node IP Solace PubSub+ software event broker 1 IP 8301 Allow TCP and UDP Solace PubSub+ software event broker 2 IP Monitoring Node IP Solace PubSub+ software event broker 1 IP 8302 Allow TCP Solace PubSub+ software event broker 2 IP Solace PubSub+ software event broker 1 IP 8741 Allow TCP Solace PubSub+ software event broker 2 IP Solace PubSub+ software event broker 1 IP 55555 Inbound Firewall Rules for Solace PubSub+ software event broker 2
Permission Protocol Source Destination Port Allow TCP Solace PubSub+ software event broker 1 IP Monitoring Node IP Solace PubSub+ software event broker 2 IP 8300 Allow TCP and UDP Solace PubSub+ software event broker 1 IP Monitoring Node IP Solace PubSub+ software event broker 2 IP 8301 Allow TCP and UDP Solace PubSub+ software event broker 1 IP Monitoring Node IP Solace PubSub+ software event broker 2 IP 8302 Allow TCP Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP 8741 Allow TCP Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP 55555 Inbound Firewall Rules for Monitoring Node
Permission Protocol Source Destination Port Allow TCP Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP Monitoring Node IP 8300 Allow TCP and UDP Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP Monitoring Node IP 8301 Allow TCP and UDP Solace PubSub+ software event broker 1 IP Solace PubSub+ software event broker 2 IP Monitoring Node IP 8302
Configuration Parameters
The following table lists parameter values that are required in the configuration procedure.
Parameter | Description | Example Values |
---|---|---|
Interface Name |
The name of the interface the Management VRF uses.
To determine the name, enter the show ip vrf management command—it is the name displayed in the Interface column without the “:<#>” suffix. |
intf0
|
IP Interface Name | The IP interface name is the interface name concatenated with a “:1”. |
|
Pre-Shared Authentication Key | 32 to 256 bytes of binary data encoded in base 64. Must be the same for all members of the HA group. |
|
IP Address |
A Solace PubSub+ software event broker's unique IP address/netmask. To determine the IP address, enter the |
Primary software event broker: 192.168.132.16/20
Backup software event broker: 192.168.132.39/20
Monitoring node: 192.168.132.3/20 |
HA redundancy group parameters |
The Solace PubSub+ software event broker name, IP address or fully qualified domain name (FQDN), and the port number for each node.
|
Primary event broker: Backup event broker: Monitoring node: |
A node role assigned for each node. |
Primary: Backup: Monitoring: Monitoring node doesn't have the |
Considerations When Using Bridge Networking
If you configure a Solace PubSub+ software event broker container to use bridge networking, the three redundancy ports (which defaults to 8300, 8301, 8302) must be published. You should ensure the following ports are accessible:
- port 8300 must be published for TCP
- ports 8301 and 8302 must be published for TCP and UDP
For example, docker create API supports the -p
flag or --publish
flag to publish ports. A typical docker create
command line would include: -p 8300:8300 -p 8301:8301 -p 8302:8302 -p 8301:8301/udp -p 8302:8302/udp
.
For more information on the use of Bridge Networking in the configuration of HA groups, refer to Port Publishing in Multi-Tenant Scenarios.
Step 2: Configure the Nodes
To configure an HA redundancy group, you must configure the primary, backup, and monitoring nodes for Direct messaging. Once all the three nodes are configured, they are effectively enabled for Direct Messaging.
To configure the Solace PubSub+ software event broker redundancy nodes, perform the following steps in order:
Before configuring Solace PubSub+ software event broker redundancy, we recommend that you first shut down the message-spool.
Configure the Primary Event Broker
To configure the primary event broker for an HA redundancy group, perform the following steps:
- On the event broker that will act as the primary (
solace1
in the examples), shut down the message spool:solace1> enable solace1# configure solace1(configure)# hardware message-spool shutdown All message spooling will be stopped. Do you want to continue (y/n)? y
- Configure the hostlist failover mechanism:
solace1(configure)# redundancy solace1(configure/redundancy)# switchover-mechanism hostlist solace1(configure/redundancy)# exit
- Configure the HA redundancy group, assign an
active-standby-role
ofprimary
to this event broker, and then enable redundancy:solace1(configure)# redundancy solace1(configure/redundancy)# group solace1(configure/redundancy/group)# create node solace1 solace1(configure/redundancy/group/node)# connect-via 192.168.132.16 solace1(configure/redundancy/group/node)# node-type message-routing-node solace1(configure/redundancy/group/node)# exit solace1(configure/redundancy/group)# create node solace2 solace1(configure/redundancy/group/node)# connect-via 192.168.132.39 solace1(configure/redundancy/group/node)# node-type message-routing-node solace1(configure/redundancy/group/node)# exit solace1(configure/redundancy/group)# create node solace3 solace1(configure/redundancy/group/node)# connect-via 192.168.132.3 solace1(configure/redundancy/group/node)# node-type monitor-node solace1(configure/redundancy/group/node)# exit solace1(configure/redundancy/group)# exit solace1(configure/redundancy)# authentication solace1(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key> solace1(configure/redundancy/authentication)# exit solace1(configure/redundancy)# active-standby-role primary solace1(configure/redundancy)# no shutdown
Where:
<pre-shared-key>
is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.In release 9.1.0 and earlier, you must use the
redundancy group password <password>
command to configure the group password for authentication between the nodes. In these releases, group passwords must be between 1 and 128 characters and are required to be the same on each node.
Configure the Backup Event Broker
To configure the backup event broker for an HA redundancy group, perform the following steps:
- On the event broker that will act as the backup (
solace2
in the examples), shut down the message spool: - Configure hostlist as the failover mechanism:
solace2(configure)# redundancy solace2(configure/redundancy)# switchover-mechanism hostlist
- Configure the HA redundancy group, assign an
active-standby-role
ofbackup
to this event broker, and then enable redundancy:solace2(configure)# redundancy solace2(configure/redundancy)# group solace2(configure/redundancy/group)# create node solace1 solace2(configure/redundancy/group/node)# connect-via 192.168.132.16 solace2(configure/redundancy/group/node)# node-type message-routing-node solace2(configure/redundancy/group/node)# exit solace2(configure/redundancy/group)# create node solace2 solace2(configure/redundancy/group/node)# connect-via 192.168.132.39 solace2(configure/redundancy/group/node)# node-type message-routing-node solace2(configure/redundancy/group/node)# exit solace2(configure/redundancy/group)# create node solace3 solace2(configure/redundancy/group/node)# connect-via 192.168.132.3 solace2(configure/redundancy/group/node)# node-type monitor-node solace2(configure/redundancy/group/node)# exit solace2(configure/redundancy/group)# exit solace2(configure/redundancy)# authentication solace2(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key> solace2(configure/redundancy/authentication)# exit solace2(configure/redundancy)# active-standby-role backup solace2(configure/redundancy)# no shutdown
Where:
<pre-shared-key>
is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.In release 9.1.0 and earlier, you must use the
redundancy group password <password>
command to configure the group password for authentication between the nodes. In these releases, group passwords must be between 1 and 128 characters and are required to be the same on each node.
solace2> enable solace2# configure solace2(configure)# hardware message-spool shutdown All message spooling will be stopped Do you want to continue (y/n)? y
Configure the Monitoring Node
To configure the monitoring node for an HA redundancy group, perform the following steps:
- On the event broker that will act as the monitoring node (
solace3
in the examples), reload the default configuration:solace3# reload default-config monitoring-node This command causes a reload of the system Do you want to continue (y/n)? y
For Solace PubSub+ software event brokers running as containers, after reloading the default configuration for the monitoring node, you must start the container manually.
- Configure the hostlist failover mechanism:
solace3(configure)# redundancy solace3(configure/redundancy)# switchover-mechanism hostlist
- Configure the HA redundancy group and enable redundancy.
Notice that as a monitoring node, this event broker doesn't have the active-standby-role assigned.
solace3(configure)# redundancy solace3(configure/redundancy)# group solace3(configure/redundancy/group)# create node solace1 solace3(configure/redundancy/group/node)# connect-via 192.168.132.16 solace3(configure/redundancy/group/node)# node-type message-routing-node solace3(configure/redundancy/group/node)# exit solace3(configure/redundancy/group)# create node solace2 solace3(configure/redundancy/group/node)# connect-via 192.168.132.39 solace3(configure/redundancy/group/node)# node-type message-routing-node solace3(configure/redundancy/group/node)# exit solace3(configure/redundancy/group)# create node solace3 solace3(configure/redundancy/group/node)# connect-via 192.168.132.3 solace3(configure/redundancy/group/node)# node-type monitor-node solace3(configure/redundancy/group/node)# exit solace3(configure/redundancy/group)# exit solace3(configure/redundancy)# authentication solace3(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key> solace3(configure/redundancy/authentication)# exit solace3(configure/redundancy)# no shutdown
Where:
<pre-shared-key>
is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.In release 9.1.0 and earlier, you must use the
redundancy group password <password>
command to configure the group password for authentication between the nodes. In these releases, group passwords must be between 1 and 128 characters and are required to be the same on each node.
Step 3: Enable Guaranteed Messaging
By default, an HA group has Guaranteed messaging disabled. It can only be enabled after the primary, backup, and monitoring nodes have redundancy configured, and although the use of Guaranteed messaging is optional, it's recommended because it's required for the group to:
- Accept clients that send or receive Guaranteed messages.
- Use Config-Sync.
Prior to release 9.2.0, mate-link connect-via [<addr-port>]
was used to start the mate-link connection. In release 9.2.0+, mate-link connect via
command has been deprecated and instead the connect-via address from the group node is used by default. In the example below, the default HA mate link port is used, if you need to use some other port in your deployment, execute the configure/redundancy/mate-link/port <port>
Solace CLI command to set the value.
To enable Guaranteed messaging for an HA group, perform the following steps on the primary and backup event brokers:
- Enable the message spool on both event brokers.
On the primary, enter the following commands:
solace1(configure)# hardware message-spool solace1(configure/hardware/message-spool)# no shutdown solace1(configure/hardware/message-spool)# exit solace1(configure/hardware)# exit
On the backup, enter the following commands:
solace2(configure)# hardware message-spool solace2(configure/hardware/message-spool)# no shutdown solace2(configure/hardware/message-spool)# exit solace2(configure/hardware)# exit
- Start the mate-link service on both event brokers.
On the primary, enter the following commands:
solace1(configure)# service mate-link solace1(configure/service/mate-link)# no shutdown solace1(configure/service/mate-link)# exit solace1(configure/service)# exit
On the backup, enter the following commands:
solace2(configure)# service mate-link solace2(configure/service/mate-link)# no shutdown solace2(configure/service/mate-link)# exit solace2(configure/service)# exit
- Optional: You can enable encryption on the mate-link.
- To verify that Guaranteed messaging for the group is enabled, use the
show redundancy
command. The values ofADB Link To Mate
,ADB Hello To Mate
, andMessage Spool Status
are what you need to look at.On the primary you should see
Up
andAD-Active
:solace1(configure/redundancy)# show redundancy Configuration Status : Enabled . . . ADB Link To Mate : Up ADB Hello To Mate : Up . . . Message Spool Status AD-Active
Likewise, on the backup you should see
Up
andAD-Standby
:solace2(configure/redundancy)# show redundancy Configuration Status : Enabled . . . ADB Link To Mate : Up ADB Hello To Mate : Up . . . Message Spool Status AD-Standby
On the primary, enter the following commands:
solace1(configure)# redundancy mate-link solace1(configure/redundancy/mate-link)# ssl solace1(configure/redundancy/mate-link)# exit solace1(configure/redundancy)# exit
On the backup, enter the following commands:
solace2(configure)# redundancy mate-link solace2(configure/redundancy/mate-link)# ssl solace2(configure/redundancy/mate-link)# exit solace2(configure/redundancy)# exit
Step 4: Enable Config-Sync
After a Solace PubSub+ software event broker HA redundancy group is configured to support Guaranteed messaging, use Config-Sync to synchronize configurations between primary and backup event brokers.
For Config-Sync to be enabled, make sure that the inbound TCP rule at port 55555 is allowed for the mate event broker to connect. See Prerequisites for more information.
Set up Config-Sync without TLS
To enable Config-Sync for the HA group, perform the following steps:
- Enable Config-Sync.
On the primary (
solace1
) event broker enter the following commands:solace1(configure)# config-sync solace1(configure/config-sync)# no shutdown solace1(configure/config-sync)# exit solace1(configure)# exit
On the backup (
solace2
) event broker enter the following commands:solace2(configure)# config-sync solace2(configure/config-sync)# no shutdown solace2(configure/config-sync)# exit solace2(configure)# exit
- Assert the primary event broker's configuration.
When enabling Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the master event broker (usually it's the primary event broker in the pair) over its mate.
As a result of this operation the
Oper Status
of Config-Sync should change fromDown
toUp
on both primary and backup event brokers (check it with theshow config-sync
User EXEC command).solace1# admin solace1(admin)# config-sync solace1(admin/config-sync)# assert-master router Processed 1 config-sync tables.
- Assert one Message VPN's configuration.
If the Message VPNs' configurations are out-of-sync, you must manually synchronize a Message VPN configuration between two event brokers. To do this, you must select one of the message VPNs as a master (it doesn't have to be on the master or primary event broker), and assert its configuration over the same Message VPN on the other event broker.
solace1(admin/config-sync)# assert-master message-vpn myvpn WARNING: This command can temporarily disconnect clients on the AD-inactive appliance. As well in-flight messages may not be delivered to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance. Do you want to continue (y/n)? y Processed 1 config-sync tables.
Set up Config-Sync over TLS
To enable Config-Sync over TLS for the HA group, perform the following steps.
All event brokers in the HA group must be configured with the same pre-shared authentication key as shown in the previous steps to be able to enable Config-Sync over TLS.
- Enable Config-Sync.
On the primary (
solace1
) event broker enter the following commands:solace1(configure)# config-sync solace1(configure/config-sync)# ssl solace1(configure/config-sync)# no shutdown solace1(configure/config-sync)# exit solace1(configure)# exit
On the backup (
solace2
) event broker enter the following commands:solace2(configure)# config-sync solace2(configure/config-sync)# ssl solace2(configure/config-sync)# no shutdown solace2(configure/config-sync)# exit solace2(configure)# exit
- Assert the primary event broker's configuration.
When enabling Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the master event broker (usually it's the primary event broker in the pair) over its mate.
As a result of this operation the
Oper Status
of Config-Sync should change fromDown
toUp
on both primary and backup event brokers (check it with theshow config-sync
User EXEC command).solace1# admin solace1(admin)# config-sync solace1(admin/config-sync)# assert-master router Processed 1 config-sync tables.
- Assert one Message VPN's configuration.
If the Message VPNs' configurations are out-of-sync, you must manually synchronize a Message VPN configuration between two event brokers. To do this, you must select one of the message VPNs as a master (it doesn't have to be on the master or primary event broker), and assert its configuration over the same Message VPN on the other event broker.
solace1(admin/config-sync)# assert-master message-vpn myvpn WARNING: This command can temporarily disconnect clients on the AD-inactive appliance. As well in-flight messages may not be delivered to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance. Do you want to continue (y/n)? y Processed 1 config-sync tables.
Step 5: Validate Failover
To validate the HA group's failover operation, perform the following steps:
- Manually release activity on the primary event broker (see Releasing Event Broker Activity), and test that messages are being published and received by the backup event broker.
solace1> enable solace1# configure solace1(configure)# redundancy solace1(configure/redundancy)# release-activity
Note that after running the
release-activity
command, the backup node will be up, but the redundancy will be down. You must run theno release-activity
, as shown in Step 3 below, to ensure the primary node rejoins the HA group, and the redundancy is back up. - Validate that all the clients successfully reconnect to the backup event broker after activity is released on the primary event broker, and test messages continue to be published and received as expected.
Use the
show stats client
command to check the client connections on the backup event broker. The total number of connected clients on the back event broker must same as it was in the primary event broker.Example :
solace2# show stats client Total Clients: 4 Total Clients Connected: 4 . . .
- Manually take the activity back to the primary event broker.
solace1(configure/redundancy)# no release-activity solace1(configure/redundancy)# home
- Force the backup event broker to give up activity (see Forcing Backups to Give Up Activity to Primaries).
solace2> enable solace2# admin solace2(admin)# redundancy solace2(admin/redundancy)# revert-activity
- Validate that all the clients reconnect back to the primary event broker after the activity is released on the backup event broker, and test that messages continue to be published and received as expected.
Use the
show stats client
command to check the client connections on the primary event broker.Example :
solace1# show stats client Total Clients: 4 Total Clients Connected: 4 . . .
Next Steps
You now have three Solace PubSub+ software event brokers configured in an HA group. You can now do things like use the SDKPerf tool to test messaging, configure an event broker's health check related settings, or configure the maximum number of client connections.
- Download SDKPerf—To get started, see SDKPerf's Quick Start guide.
- Load Balancer Health Checks—If the Solace PubSub+ software event brokers are operating in conjunction with an environment provided load balancer, it is recommended to review the Load Balancer Health Checks feature page.
- Setting Scaling Parameters for an HA Group—Configure system scaling parameters for HA redundancy groups using the Solace CLI.