HA Group Configuration

In this section you'll learn how to set up a high-availability (HA) redundancy group of Solace PubSub+ software event brokers.

The steps shown in this topic are generally applicable to HA group configuration, but if you need to set up an HA group on a Windows or macOS platform, instructions tailored for doing just that can be found on these pages:

As well, if you're interested in setting up an HA group in AWS, over in GitHub there's a Quick Start called

that will take you through the steps of HA configuration.

Although the example shown in this topic uses Solace CLI, the intent is to help you become familiar with the ins-and-outs of HA configuration as a step towards using more advanced, production-oriented techniques. For instance, you can also get an HA group up-and-running using configuration keys, and if you're interested, you may want to jump ahead to Configuring HA Groups Using Configuration Keys.

Step 1: Review Prerequisites & Configuration Parameters

Before configuring an HA redundancy group, you need to review the following:

Prerequisites

The configuration steps provided in this section assumes that:

  • You have obtained a Solace PubSub+ software event broker package. To download a copy of a software event broker package, visit Solace PubSub+ software event broker downloads page.
  • The software event brokers to be used in the HA redundancy group use:
    • Configuration defaults.
    • Host networking and single tenant hosts.

With regard to the above assumptions, the only configuration changes required are the following:

  1. Each software event broker in the HA redundancy group is configured with a unique host and router name (see Hostname Configuration).

  2. Each software event broker in the HA redundancy group is configured with a unique IP address associated with the Management VRF (see Software Event Broker Host Interface Configuration).
  3. All software event brokers in the HA redundancy group must have the same release version.
  4. All messaging nodes within the HA group configured to use the same connection scaling tier. When upgrading a monitoring node of an HA group to Solace PubSub+ version 8.10 or higher, the database is transformed to use the 100 connection scaling tier (see Scaling Tiers for Software Event Brokers).
  5. System clocks in both the primary and backup event brokers are synchronized. with a Network Time Protocol (NTP) server (see the Software Event Brokers section of Synchronizing Clocks with NTP Servers).
  6. Each software event broker in the HA redundancy group is deployed on a different physical host system.
  7. The primary and backup software event brokers are run in identical virtual machines (with the same amount of virtual RAM and same number of virtual processor cores) and use the same size disks for their message spools.
  8. If you are deploying the HA redundancy group in a cloud environment, you must ensure that security rules are created for ports 8300, 8301, 8302 (the three HA configuration synchronization ports), 8741 (the HA mate link port), and 55555 (the SMF port). See the firewall rule tables, shown below, for details.
  9. In the example discussed in this section, the default HA mate link port is used. If you need to use some other port in your deployment, you'll need to use the configure/redundancy/mate-link/port <port> Solace CLI command to set the value. For more information about the command, you can look it up in the Software Event Broker CLI Commands.

    Inbound Firewall Rules for Solace PubSub+ software event broker 1

    PermissionProtocolSourceDestinationPort
    AllowTCPSolace PubSub+ software event broker 2 IP
    Monitoring Node IP    		Solace PubSub+ software event broker 1 IP8300
    AllowTCP and UDPSolace PubSub+ software event broker 2 IP
    Monitoring Node IP    		Solace PubSub+ software event broker 1 IP8301
    AllowTCP and UDPSolace PubSub+ software event broker 2 IP
    Monitoring Node IP    		Solace PubSub+ software event broker 1 IP8302
    AllowTCP Solace PubSub+ software event broker 2 IPSolace PubSub+ software event broker 1 IP8741
    AllowTCPSolace PubSub+ software event broker 2 IPSolace PubSub+ software event broker 1 IP55555

    Inbound Firewall Rules for Solace PubSub+ software event broker 2

    PermissionProtocolSourceDestinationPort
    AllowTCPSolace PubSub+ software event broker 1 IP
    Monitoring Node IP    		Solace PubSub+ software event broker 2 IP8300
    AllowTCP and UDPSolace PubSub+ software event broker 1 IP
    Monitoring Node IP    		Solace PubSub+ software event broker 2 IP8301
    AllowTCP and UDPSolace PubSub+ software event broker 1 IP
    Monitoring Node IP    		Solace PubSub+ software event broker 2 IP8302
    AllowTCP Solace PubSub+ software event broker 1 IPSolace PubSub+ software event broker 2 IP8741
    AllowTCPSolace PubSub+ software event broker 1 IPSolace PubSub+ software event broker 2 IP55555

    Inbound Firewall Rules for Monitoring Node

    PermissionProtocolSourceDestinationPort
    AllowTCPSolace PubSub+ software event broker 1 IP
    Solace PubSub+ software event broker 2 IP    		Monitoring Node IP8300
    AllowTCP and UDPSolace PubSub+ software event broker 1 IP
    Solace PubSub+ software event broker 2 IP    		Monitoring Node IP8301
    AllowTCP and UDPSolace PubSub+ software event broker 1 IP
    Solace PubSub+ software event broker 2 IP    		Monitoring Node IP8302

Configuration Parameters

The following table lists parameter values that are required in the configuration procedure.

Parameter Description Example Values
Interface Name The name of the interface the Management VRF uses. To determine the name, enter the show ip vrf management command—it is the name displayed in the Interface column without the “:<#>” suffix. intf0
IP Interface Name The IP interface name is the interface name concatenated with a “:1”.

intf0:1

Pre-Shared Authentication Key 32 to 256 bytes of binary data encoded in base 64. Must be the same for all members of the HA group.

MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDEK

IP Address

A Solace PubSub+ software event broker’s unique IP address/netmask.

To determine the IP address, enter the show ip vrf management command—it is the IP address displayed in the output for the Static V Router.

 Primary software event broker: 192.168.132.16/20
Backup software event broker: 192.168.132.39/20
Monitoring node: 192.168.132.3/20
HA redundancy group parameters

The Solace PubSub+ software event broker name, IP address or fully qualified domain name (FQDN), and the port number for each node.

Note:  

  • The node name must be the same as the router name.
  • Redundancy requires three ports, which must be accessible by all nodes in a redundancy group. Port 8300 is set by default, and the two subsequent ports are used as required.

Primary event broker: solace1, connect-via 192.168.132.16

Backup event broker: solace2, connect-via 192.168.132.39

Monitoring node: solace3, connect-via 192.168.132.3
A node role assigned for each node.

Primary: active-standby-role primary

Backup: active-standby-role standby

Monitoring: Monitoring node doesn’t have the active-standby-role assigned.

Considerations When Using Bridge Networking

If you configure a Solace PubSub+ software event broker container to use bridge networking, the three redundancy ports (which defaults to 8300, 8301, 8302) must be published. You should ensure the following ports are accessible:

  • port 8300 must be published for TCP
  • ports 8301 and 8302 must be published for TCP and UDP

For example, docker create API supports the -p flag or --publish flag to publish ports. A typical docker create command line would include: -p 8300:8300 -p 8301:8301 -p 8302:8302 -p 8301:8301/udp -p 8302:8302/udp.

For more information on the use of Bridge Networking in the configuration of HA groups, refer to Port Publishing in Multi-Tenant Scenarios.

Step 2: Configure the Nodes

To configure an HA redundancy group, you must configure the primary, backup, and monitoring nodes for Direct messaging. Once all the three nodes are configured, they are effectively enabled for Direct Messaging.

To configure the Solace PubSub+ software event broker redundancy nodes, perform the following steps in order:

  1. Configure the Primary Event Broker
  2. Configure the Backup Event Broker
  3. Configure the Monitoring Node

:  Before configuring Solace PubSub+ software event broker redundancy, we recommend that you first shutdown the message-spool.

Configure the Primary Event Broker

To configure the primary event broker for an HA redundancy group, perform the following steps:

  1. On the event broker that will act as the primary (solace1 in the examples), shut down the message spool:

    solace1> enable
    solace1# configure
    solace1(configure)# hardware message-spool shutdown
    All message spooling will be stopped.
    Do you want to continue (y/n)? y

  2. Configure the hostlist failover mechanism:

    solace1(configure)# redundancy
    solace1(configure/redundancy)# switchover-mechanism hostlist
    solace1(configure/redundancy)# exit

  3. Configure the HA redundancy group, assign an active-standby-role of primary to this event broker, and then enable redundancy:

    solace1(configure)# redundancy
    solace1(configure/redundancy)# group
    solace1(configure/redundancy/group)# create node solace1
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace1(configure/redundancy/group/node)# node-type message-routing-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# create node solace2
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace1(configure/redundancy/group/node)# node-type message-routing-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# create node solace3
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace1(configure/redundancy/group/node)# node-type monitor-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# exit
    solace1(configure/redundancy)# authentication
    solace1(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
    solace1(configure/redundancy/authentication)# exit
    solace1(configure/redundancy)# active-standby-role primary
    solace1(configure/redundancy)# no shutdown

    Where:

    <pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

Configure the Backup Event Broker

To configure the backup event broker for an HA redundancy group, perform the following steps:

  1. On the event broker that will act as the backup (solace2 in the examples), shut down the message spool:

    2. solace2> enable
    solace2# configure
    solace2(configure)# hardware message-spool shutdown
    All message spooling will be stopped
    Do you want to continue (y/n)? y

  2. Configure hostlist as the failover mechanism:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# switchover-mechanism hostlist

  3. Configure the HA redundancy group, assign an active-standby-role of backup to this event broker, and then enable redundancy:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# group
    solace2(configure/redundancy/group)# create node solace1
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace2(configure/redundancy/group/node)# node-type message-routing-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# create node solace2
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace2(configure/redundancy/group/node)# node-type message-routing-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# create node solace3
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace2(configure/redundancy/group/node)# node-type monitor-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# exit
    solace2(configure/redundancy)# authentication
    solace2(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
    solace2(configure/redundancy/authentication)# exit
    solace2(configure/redundancy)# active-standby-role backup
    solace2(configure/redundancy)# no shutdown

    Where:

    <pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

Configure the Monitoring Node

To configure the monitoring node for an HA redundancy group, perform the following steps:

  1. On the event broker that will act as the monitoring node (solace3 in the examples), reload the default configuration:

    solace3# reload default-config monitoring-node
    This command causes a reload of the system
    Do you want to continue (y/n)? y

    • For Solace PubSub+ software event brokers running as Docker images, after reloading the default configuration for the monitoring node, you must start the Docker container manually.

    • To change the IP address of the monitoring node (if required), see IP Addressing in Hypervisor Environments.

  2. Configure the hostlist failover mechanism:

    solace3(configure)# redundancy
    solace3(configure/redundancy)# switchover-mechanism hostlist

  3. Configure the HA redundancy group and enable redundancy.

    Notice that as a monitoring node, this event broker doesn’t have the active-standby-role assigned.

    solace3(configure)# redundancy
    solace3(configure/redundancy)# group
    solace3(configure/redundancy/group)# create node solace1
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace3(configure/redundancy/group/node)# node-type message-routing-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# create node solace2
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace3(configure/redundancy/group/node)# node-type message-routing-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# create node solace3
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace3(configure/redundancy/group/node)# node-type monitor-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# exit
    solace3(configure/redundancy)# authentication
    solace3(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
    solace3(configure/redundancy/authentication)# exit
    solace3(configure/redundancy)# no shutdown

    Where:

    <pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

Step 3: Enable Guaranteed Messaging

By default, an HA group has Guaranteed messaging disabled. It can only be enabled after the primary, backup, and monitoring nodes have redundancy configured, and although the use of Guaranteed messaging is optional, it's recommended because it's required for the group to:

  • Accept clients that send or receive Guaranteed messages.
  • Use Config-Sync.

Prior to release 9.2.0, mate-link connect-via [<addr-port>] was used to start the mate-link connection. In release 9.2.0+, mate-link connect via command has been deprecated and instead the connect-via address from the group node is used by default. In the example below, the default HA mate link port is used, if you need to use some other port in your deployment, execute the configure/redundancy/mate-link/port <port> Solace CLI command to set the value.

To enable Guaranteed messaging for an HA group, perform the following steps on the primary and backup event brokers:

  1. Enable the message spool on both event brokers.

    On the primary, enter the following commands:

    solace1(configure)# hardware message-spool
    solace1(configure/hardware/message-spool)# no shutdown
    solace1(configure/hardware/message-spool)# exit
    solace1(configure/hardware)# exit

    On the backup, enter the following commands:

    solace2(configure)# hardware message-spool
    solace2(configure/hardware/message-spool)# no shutdown
    solace2(configure/hardware/message-spool)# exit
    solace2(configure/hardware)# exit

  2. Start the mate-link service on both event brokers.

    On the primary, enter the following commands:

    solace1(configure)# service mate-link
    solace1(configure/service/mate-link)# no shutdown
    solace1(configure/service/mate-link)# exit
    solace1(configure/service)# exit

    On the backup, enter the following commands:

    solace2(configure)# service mate-link
    solace2(configure/service/mate-link)# no shutdown
    solace2(configure/service/mate-link)# exit
    solace2(configure/service)# exit

  3. Optional: You can enable encryption on the mate-link.

    4. On the primary, enter the following commands:

    solace1(configure)# redundancy mate-link
    solace1(configure/redundancy/mate-link)# ssl
    solace1(configure/redundancy/mate-link)# exit
    solace1(configure/redundancy)# exit

    On the backup, enter the following commands:

    solace2(configure)# redundancy mate-link
    solace2(configure/redundancy/mate-link)# ssl
    solace2(configure/redundancy/mate-link)# exit
    solace2(configure/redundancy)# exit

  4. To verify that Guaranteed messaging for the group is enabled, use the show redundancy command. The values of ADB Link To Mate, ADB Hello To Mate, and Message Spool Status are what you need to look at.
    • On the primary you should see Up and AD-Active:

      solace1(configure/redundancy)# show redundancy
      Configuration Status     : Enabled
      . . . 
      ADB Link To Mate         : Up
      ADB Hello To Mate        : Up
      . . . 
      Message Spool Status           AD-Active
      . . . 

    • Likewise, on the backup you should see Up and AD-Standby:

      solace2(configure/redundancy)# show redundancy
      Configuration Status     : Enabled
      . . . 
      ADB Link To Mate         : Up
      ADB Hello To Mate        : Up
      . . . 
      Message Spool Status           AD-Standby
      . . . 

Step 4: Enable Config-Sync

After a Solace PubSub+ software event broker HA redundancy group is configured to support Guaranteed messaging, use Config-Sync to synchronize configurations between primary and backup event brokers.

:  For Config-Sync to be enabled, make sure that the inbound TCP rule at port 55555 is allowed for the mate event broker to connect. See Prerequisites for more information.

Set up Config-Sync without TLS

To enable Config-Sync for the HA group, perform the following steps:

  1. Enable Config-Sync.

    On the primary (solace1) event broker enter the following commands:

    solace1(configure)# config-sync
    solace1(configure/config-sync)# no shutdown
    solace1(configure/config-sync)# exit
    solace1(configure)# exit

    On the backup (solace2) event broker enter the following commands:

    solace2(configure)# config-sync
    solace2(configure/config-sync)# no shutdown
    solace2(configure/config-sync)# exit
    solace2(configure)# exit

  2. Assert the primary event broker’s configuration.

    When enabling Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the master event broker (usually it’s the primary event broker in the pair) over its mate.

    As a result of this operation the Oper Status of Config-Sync should change from Down to Up on both primary and backup event brokers (check it with the show config-sync User EXEC command).

    solace1# admin
    solace1(admin)# config-sync
    solace1(admin/config-sync)# assert-master router
    Processed 1 config-sync tables.

  3. Assert one Message VPN’s configuration.

    If the Message VPNs’ configurations are out-of-sync, you must manually synchronize a Message VPN configuration between two event brokers. To do this, you must select one of the message VPNs as a master (it doesn’t have to be on the master or primary event broker), and assert its configuration over the same Message VPN on the other event broker.

    solace1(admin/config-sync)# assert-master message-vpn myvpn
    WARNING: This command can temporarily disconnect clients on the
    AD-inactive appliance. As well in-flight messages may not be delivered to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance.
    Do you want to continue (y/n)? y
    Processed 1 config-sync tables.

Set up Config-Sync over TLS

To enable Config-Sync over TLS for the HA group, perform the following steps.

:  All event brokers in the HA group must be configured with the same pre-shared authentication key as shown in the previous steps to be able to enable Config-Sync over TLS.

  1. Enable Config-Sync.

    On the primary (solace1) event broker enter the following commands:

    solace1(configure)# config-sync
    solace1(configure/config-sync)# ssl
    solace1(configure/config-sync)# no shutdown
    solace1(configure/config-sync)# exit
    solace1(configure)# exit

    On the backup (solace2) event broker enter the following commands:

    solace2(configure)# config-sync
    solace2(configure/config-sync)# ssl
    solace2(configure/config-sync)# no shutdown
    solace2(configure/config-sync)# exit
    solace2(configure)# exit

  2. Assert the primary event broker’s configuration.

    When enabling Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the master event broker (usually it’s the primary event broker in the pair) over its mate.

    As a result of this operation the Oper Status of Config-Sync should change from Down to Up on both primary and backup event brokers (check it with the show config-sync User EXEC command).

    solace1# admin
    solace1(admin)# config-sync
    solace1(admin/config-sync)# assert-master router
    Processed 1 config-sync tables.

  3. Assert one Message VPN’s configuration.

    If the Message VPNs’ configurations are out-of-sync, you must manually synchronize a Message VPN configuration between two event brokers. To do this, you must select one of the message VPNs as a master (it doesn’t have to be on the master or primary event broker), and assert its configuration over the same Message VPN on the other event broker.

    solace1(admin/config-sync)# assert-master message-vpn myvpn
    WARNING: This command can temporarily disconnect clients on the
    AD-inactive appliance. As well in-flight messages may not be delivered to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance.
    Do you want to continue (y/n)? y
    Processed 1 config-sync tables.

Step 5: Validate Failover

To validate the HA group’s failover operation, perform the following steps:

  1. Manually release activity on the primary event broker (see Releasing Event Broker Activity), and test that messages are being published and received by the backup event broker.

    solace1> enable
    solace1# configure
    solace1(configure)# redundancy
    solace1(configure/redundancy)# release-activity

    Note that after running the release-activity command, the backup node will be up, but the redundancy will be down. You must run the no release-activity, as shown in Step 3 below, to ensure the primary node rejoins the HA group, and the redundancy is back up.

  2. Validate that all the clients successfully reconnect to the backup event broker after activity is released on the primary event broker, and test messages continue to be published and received as expected.

    Use the show stats client command to check the client connections on the backup event broker. The total number of connected clients on the back event broker must same as it was in the primary event broker.

    solace2# show stats client
    Total Clients:            4
    Total Clients Connected:  4
    ...

  3. Manually take the activity back to the primary event broker.

    solace1(configure/redundancy)# no release-activity
    solace1(configure/redundancy)# home

  4. Force the backup event broker to give up activity (see Forcing Backups to Give Up Activity to Primaries).

    solace2> enable
    solace2# admin
    solace2(admin)# redundancy
    solace2(admin/redundancy)# revert-activity

  5. Validate that all the clients reconnect back to the primary event broker after the activity is released on the backup event broker, and test that messages continue to be published and received as expected.

    Use the show stats client command to check the client connections on the primary event broker.

    solace1# show stats client
    Total Clients:            4
    Total Clients Connected:  4
    ...

Next Steps

You now have three Solace PubSub+ software event brokers configured in an HA group. You can now do things like use the SDKPerf tool to test messaging, configure an event broker's health check related settings, or configure the connection scaling tier.