Products  Company  Blog  Support  Contact Us

HA Group Configuration

In this section you'll learn how to set up a high-availability (HA) redundancy group of Solace PubSub+ software message brokers.

The steps shown in this topic are generally applicable to HA group configuration, but if you need to set up an HA group on a Windows or macOS platform, instructions tailored for doing just that can be found on these pages,

As well, if you're interested in setting up an HA group in AWS, over in GitHub there is a quick start called

that will take you through the steps of HA configuration.

Although the example shown in this topic uses Solace CLI, the intent is to help you become familiar with the ins-and-outs of HA configuration as a step towards using more advanced, production-oriented techniques. For instance, you can also get an HA group up-and-running using configuration keys, and if you are interested, you may want to jump ahead to Configuring HA Groups Using Configuration Keys.

Step 1: Review Prerequisites & Configuration Parameters

Before configuring an HA redundancy group, you need to review the following:

Prerequisites

The configuration steps provided in this section assumes that:

  • you have obtained a Solace PubSub+ software message broker package. To download a copy of a software message broker package, visit Solace PubSub+ software message broker downloads page.
  • the software message brokers to be used in the HA redundancy group use:
    • Configuration defaults.
    • Host networking and single tenant hosts.

With regard to the above assumptions, the only configuration changes required are the following:

  1. Each software message broker in the HA redundancy group is configured with a unique host and router name (see Hostname Configuration).

  2. Each software message broker in the HA redundancy group is configured with a unique IP address associated with the Management VRF (see IP Addressing in Hypervisor Environments).
  3. All software message brokers in the HA redundancy group must have the same release version.
  4. All messaging nodes within the HA group configured to use the same connection scaling tier. When upgrading a monitoring node of an HA group to Solace PubSub+ version 8.10 or higher, the database is transformed to use the 100 connection scaling tier (see Scaling Tiers for Software Message Brokers).
  5. System clocks in both primary and backup message brokers are synchronized with a Network Time Protocol (NTP) server (see Synchronizing Clocks with NTP Servers).
  6. Each software message broker in the HA redundancy group is deployed on a different physical host system.
  7. The primary and backup software message brokers are run in identical virtual machines (with the same amount of virtual RAM and same number of virtual processor cores) and use the same size disks for their message spools.
  8. If you are deploying the HA redundancy group in a cloud environment, you must ensure that security rules are created for ports 8300, 8301, 8302 (the three HA configuration synchronization ports), 8741 (the HA mate link port), and 55555 (the SMF port). See the following tables for details:

    Inbound Firewall Rules for Solace PubSub+ software message broker 1

    PermissionProtocolSourceDestinationPort
    AllowTCPSolace PubSub+ software message broker 2 IP
    Monitoring Node IP    		Solace PubSub+ software message broker 1 IP8300
    AllowTCP and UDPSolace PubSub+ software message broker 2 IP
    Monitoring Node IP    		Solace PubSub+ software message broker 1 IP8301
    AllowTCP and UDPSolace PubSub+ software message broker 2 IP
    Monitoring Node IP    		Solace PubSub+ software message broker 1 IP8302
    AllowTCP Solace PubSub+ software message broker 2 IPSolace PubSub+ software message broker 1 IP8741
    AllowTCPSolace PubSub+ software message broker 2 IPSolace PubSub+ software message broker 1 IP55555

    Inbound Firewall Rules for Solace PubSub+ software message broker 2

    PermissionProtocolSourceDestinationPort
    AllowTCPSolace PubSub+ software message broker 1 IP
    Monitoring Node IP    		Solace PubSub+ software message broker 2 IP8300
    AllowTCP and UDPSolace PubSub+ software message broker 1 IP
    Monitoring Node IP    		Solace PubSub+ software message broker 2 IP8301
    AllowTCP and UDPSolace PubSub+ software message broker 1 IP
    Monitoring Node IP    		Solace PubSub+ software message broker 2 IP8302
    AllowTCP Solace PubSub+ software message broker 1 IPSolace PubSub+ software message broker 2 IP8741
    AllowTCPSolace PubSub+ software message broker 1 IPSolace PubSub+ software message broker 2 IP55555

    Inbound Firewall Rules for Monitoring Node

    PermissionProtocolSourceDestinationPort
    AllowTCPSolace PubSub+ software message broker 1 IP
    Solace PubSub+ software message broker 2 IP    		Monitoring Node IP8300
    AllowTCP and UDPSolace PubSub+ software message broker 1 IP
    Solace PubSub+ software message broker 2 IP    		Monitoring Node IP8301
    AllowTCP and UDPSolace PubSub+ software message broker 1 IP
    Solace PubSub+ software message broker 2 IP    		Monitoring Node IP8302

Configuration Parameters

The following table lists parameter values that are required in the configuration procedure.

Parameter Description Example Values
Interface Name The name of the interface the Management VRF uses. To determine the name, enter the show ip vrf management command—it is the name displayed in the Interface column without the “:<#>” suffix. intf0
IP Interface Name The IP interface name is the interface name concatenated with a “:1”.

intf0:1

IP Address

A Solace PubSub+ software message broker’s unique IP address/netmask.

To determine the IP address, enter the show ip vrf management command—it is the IP address displayed in the output for the Static V Router.

 Primary software message broker: 192.168.132.16/20
Backup software message broker: 192.168.132.39/20
Monitoring node: 192.168.132.3/20
HA redundancy group parameters

The Solace PubSub+ software message broker name, IP address or fully qualified domain name (FQDN), and the port number for each node.

Note:  

  • The node name must be the same as the router name.
  • Redundancy requires three ports, which must be accessible by all nodes in a redundancy group. Port 8300 is set by default, and the two subsequent ports are used as required.

Primary message broker: solace1, connect-via 192.168.132.16

Backup message broker: solace2, connect-via 192.168.132.39

Monitoring node: solace3, connect-via 192.168.132.3
A node role assigned for each node.

Primary: active-standby-role primary

Backup: active-standby-role standby

Monitoring: Monitoring node doesn’t have the active-standby-role assigned.

Considerations When Using Bridge Networking

If a Solace PubSub+ software message broker Docker Container is configured to use bridge networking, ports 8300, 8301, 8302 must be published so that they are mapped 1-to-1 with host ports. For example, the docker create command line must include: -p 8300:8300 -p 8301:8301 -p 8302:8302.

For more information on the use of Bridge Networking in the configuration of HA groups, refer to Port Publishing in Multi-Tenant Scenarios.

Step 2: Configure the Nodes

To configure an HA redundancy group, you must configure the primary, backup, and monitoring nodes for Direct messaging. Once all the three nodes are configured, they are effectively enabled for Direct Messaging.

To configure the Solace PubSub+ software message broker redundancy nodes, perform the following steps in order:

  1. Configure the Primary Message Broker
  2. Configure the Backup Message Broker
  3. Configure the Monitoring Node

Tip:  Before configuring Solace PubSub+ software message broker redundancy, Solace recommends that you first shutdown the message-spool.

Configure the Primary Message Broker

To configure the primary message broker for an HA redundancy group, perform the following steps:

  1. On the message broker that will act as the primary (solace1 in the examples), shut down the message spool:

    solace1> enable
    solace1# configure
    solace1(configure)# hardware message-spool shutdown
    All message spooling will be stopped.
    Do you want to continue (y/n)? y

  2. Configure the hostlist failover mechanism:

    solace1(configure)# redundancy
    solace1(configure/redundancy)# switchover-mechanism hostlist
    solace1(configure/redundancy)# exit

  3. Configure the HA redundancy group, assign an active-standby-role of primary to this message broker, and then enable redundancy:

    solace1(configure)# redundancy
    solace1(configure/redundancy)# group
    solace1(configure/redundancy/group)# create node solace1
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace1(configure/redundancy/group/node)# node-type message-routing-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# create node solace2
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace1(configure/redundancy/group/node)# node-type message-routing-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# create node solace3
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace1(configure/redundancy/group/node)# node-type monitor-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# password <password>
    solace2(configure/redundancy/group)# exit
    solace1(configure/redundancy)# active-standby-role primary
    solace1(configure/redundancy)# no shutdown

    Where:

    password is the password used for authentication between nodes in a HA Group. This password must be the same on each node in the HA Group.

Configure the Backup Message Broker

To configure the backup message broker for an HA redundancy group, perform the following steps:

  1. On the message broker that will act as the backup (solace2 in the examples), shut down the message spool:

    2. solace2> enable
    solace2# configure
    solace2(configure)# hardware message-spool shutdown
    All message spooling will be stopped
    Do you want to continue (y/n)? y

  2. Configure hostlist as the failover mechanism:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# switchover-mechanism hostlist

  3. Configure the HA redundancy group, assign an active-standby-role of backup to this message broker, and then enable redundancy:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# group
    solace2(configure/redundancy/group)# create node solace1
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace2(configure/redundancy/group/node)# node-type message-routing-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# create node solace2
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace2(configure/redundancy/group/node)# node-type message-routing-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# create node solace3
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace2(configure/redundancy/group/node)# node-type monitor-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# password <password>
    solace2(configure/redundancy/group)# exit
    solace2(configure/redundancy)# active-standby-role backup
    solace2(configure/redundancy)# no shutdown

    Where:

    password is the password used for authentication between nodes in a HA Group. This password must be the same on each node in the HA Group.

Configure the Monitoring Node

To configure the monitoring node for an HA redundancy group, perform the following steps:

  1. On the message broker that will act as the monitoring node (solace3 in the examples), reload the default configuration:

    solace3# reload default-config monitoring-node
    This command causes a reload of the system
    Do you want to continue (y/n)? y

    Note  

    • For Solace PubSub+ software message brokers running as Docker images, after reloading the default configuration for the monitoring node, you must start the Docker container manually.

    • To change the IP address of the monitoring node (if required), see IP Addressing in Hypervisor Environments.

  2. Configure the hostlist failover mechanism:

    solace3(configure)# redundancy
    solace3(configure/redundancy)# switchover-mechanism hostlist

  3. Configure the HA redundancy group and enable redundancy.

    Notice that as a monitoring node, this message broker doesn’t have the active-standby-role assigned.

    solace3(configure)# redundancy
    solace3(configure/redundancy)# group
    solace3(configure/redundancy/group)# create node solace1
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace3(configure/redundancy/group/node)# node-type message-routing-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# create node solace2
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace3(configure/redundancy/group/node)# node-type message-routing-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# create node solace3
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace3(configure/redundancy/group/node)# node-type monitor-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# password <password>
    solace2(configure/redundancy/group)# exit
    solace3(configure/redundancy)# no shutdown

    Where:

    password is the password used for authentication between nodes in a HA Group. This password must be the same on each node in the HA Group.

Step 3: Set Up Guaranteed Messaging

By default, an HA redundancy group does not have Guaranteed messaging enabled. Guaranteed messaging can only be enabled after primary, backup, and monitoring nodes have redundancy configured. Although Guaranteed messaging is optional, it is recommended because Guaranteed messaging is required for the group to:

  • accept clients that send or receive Guaranteed messages
  • use the recommended Config-Sync functionality

To enable Guaranteed Messaging for the Solace PubSub+ software message broker Redundancy Group, perform the following steps:

  1. Enable Guaranteed Messaging
  2. Enable Config-Sync

Enable Guaranteed Messaging

To enable Guaranteed messaging for a Solace PubSub+ software message broker HA redundancy group, perform the following steps for the primary and backup message brokers:

  1. Configure the mate link connect-via parameter on both primary and backup message brokers.

    The connect-via parameter value for each message broker must be the IP address or FQDN of its mate. That is, the primary message broker must point to the IP address or FQDN of the backup message broker, and the backup message broker must point to the IP address or FQDN of the primary message broker as shown below.

    Linking with Connect-Via Property

    Guaranteed Messaging Connect-Via Property

    On the primary message broker, enter the following commands :

    solace1(configure)# redundancy
    solace1(configure/redundancy)# shutdown
    solace1(configure/redundancy)# mate-link connect-via 192.168.132.39
    solace1(configure/redundancy)# no shutdown
    solace1(configure/redundancy)# exit

    On the backup message broker, enter the following commands:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# shutdown
    solace2(configure/redundancy)# mate-link connect-via 192.168.132.16
    solace2(configure/redundancy)# no shutdown
    solace2(configure/redundancy)# exit

  2. Enable the message spool on both message brokers.

    On the primary message broker, enter the following commands:

    solace1(configure)# hardware message-spool
    solace1(configure/hardware/message-spool)# no shutdown
    solace1(configure/hardware/message-spool)# exit
    solace1(configure/hardware)# exit

    On the backup message broker, enter the following commands:

    solace2(configure)# hardware message-spool
    solace2(configure/hardware/message-spool)# no shutdown
    solace2(configure/hardware/message-spool)# exit
    solace2(configure/hardware)# exit

  3. Start the mate-link service on both message brokers.

    On the primary message broker, enter the following commands:

    solace1(configure)# service mate-link
    solace1(configure/service/mate-link)# no shutdown
    solace1(configure/service/mate-link)# exit
    solace1(configure/service)# exit

    On the backup message broker, enter the following commands:

    solace2(configure)# service mate-link
    solace2(configure/service/mate-link)# no shutdown
    solace2(configure/service/mate-link)# exit
    solace2(configure/service)# exit

  4. To verify that Guaranteed Messaging for the group becomes enabled, use the show redundancy command. The values of ADB Link To Mate, ADB Hello To Mate, and Message Spool Status indicate that the Guaranteed Messaging is enabled.
    • The primary message broker should have Up and AD-Active values:

      solace1(configure/redundancy)# show redundancy
      Configuration Status     : Enabled
      . . . 
      ADB Link To Mate         : Up
      ADB Hello To Mate        : Up
      . . . 
      Message Spool Status           AD-Active
      . . . 

    • The backup message broker should have Up and AD-Standby values:

      solace2(configure/redundancy)# show redundancy
      Configuration Status     : Enabled
      . . . 
      ADB Link To Mate         : Up
      ADB Hello To Mate        : Up
      . . . 
      Message Spool Status           AD-Standby
      . . . 

Enable Config-Sync

After a Solace PubSub+ software message broker HA redundancy group is configured to support Guaranteed messaging, use Config-Sync to synchronize configurations between primary and backup message brokers.

Note:  For config-sync to be enabled, make sure that the inbound TCP rule at port 55555 is allowed for the mate message broker to connect. See Prerequisites for more information.

To enable Config-Sync for the HA group, perform the following steps:

  1. Enable Config-Sync.

    On the primary (solace1) message broker enter the following commands:

    solace1(configure)# config-sync
    solace1(configure/config-sync)# no shutdown
    solace1(configure/config-sync)# exit
    solace1(configure)# exit

    On the backup (solace2) message broker enter the following commands:

    solace2(configure)# config-sync
    solace2(configure/config-sync)# no shutdown
    solace1(configure/config-sync)# exit
    solace1(configure)# exit

  2. Assert the primary message broker’s configuration.

    When enabling the Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the master message broker (usually it’s the primary message broker in the pair) over its mate.

    As a result of this operation the Open Status of Config-Sync should change from Down to Up on both primary and backup message brokers (check it with the show config-sync User EXEC command).

    solace1# admin
    solace1(admin)# config-sync
    solace1(admin/config-sync)# assert-master router
    Processed 1 config-sync tables.

  3. Assert one Message VPN’s configuration.

    If the Message VPNs’ configurations are out of sync, you must manually synchronize a Message VPN configuration between two message brokers. To do this, you must select one of the message VPNs as a master (it doesn’t have to be on the master or primary message broker), and assert its configuration over the same Message VPN on the other message broker.

    solace1(admin/config-sync)# assert-master message-vpn myvpn
    WARNING: This command can temporarily disconnect clients on the
    AD-inactive appliance. As well in-flight messages may not be delivered to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance.
    Do you want to continue (y/n)? y
    Processed 1 config-sync tables.

Step 4: Validate Failover

To validate the HA group’s failover operation, perform the following steps:

  1. Manually release activity on the primary message broker (see Releasing Message Broker Activity), and test that messages are being published and received by the backup message broker.

    solace1> enable
    solace1# configure
    solace1(configure)# redundancy
    solace1(configure/redundancy)# release-activity

  2. Validate that all the clients successfully reconnect to the backup message broker after activity is released on the primary message broker, and test messages continue to be published and received as expected.

    Use the show stats client command to check the client connections on the backup message broker. The total number of connected clients on the back message broker must same as it was in the primary message broker.

    ClosedExample :

    solace2# show stats client
    Total Clients:            4
    Total Clients Connected:  4
    ...

  3. Manually take the activity back to the primary message broker.

    solace1(configure/redundancy)# no release-activity
    solace1(configure/redundancy)# home

  4. Force the backup message broker to give up activity (see Forcing Backups to Give Up Activity to Primaries).

    solace2> enable
    solace2# admin
    solace2(admin)# redundancy
    solace2(admin/redundancy)# revert-activity

  5. Validate that all the clients reconnect back to the primary message broker after the activity is released on the backup message broker, and test that messages continue to be published and received as expected.

    Use the show stats client command to check the client connections on the primary message broker.

    ClosedExample :

    solace1# show stats client
    Total Clients:            4
    Total Clients Connected:  4
    ...

Next Steps

You now have three Solace PubSub+ software message brokers configured in an HA group. You can now do things like use the SDKPerf tool to test messaging, configure a message broker's health check related settings, or configure the connection scaling tier.

Documentation Set: 2019.05.10.0001
Last Updated: May 10, 2019
Copyright © Solace Corporation

FacebookLinkedInTwitterYouTubeInstagramMedium