Configuring HA Redundancy Groups

To configure VMRs in an high-availability (HA) redundancy group, you must perform the following steps:

Step 1: Review Prerequisites and Configuration Parameters

Step 2: Configure VMR Redundancy Nodes

Step 3: Set Up Guaranteed Messaging

Step 4: Validate Failover

Step 1: Review Prerequisites and Configuration Parameters

Before configuring an HA redundancy group, review the Prerequisites and Configuration Parameters that are required.

Prerequisites

The configuration steps provided in this section assume that VMRs to be used in the HA redundancy group use the VMR configuration defaults. The only configuration changes that are required are the following:

  1. Each VMR in the HA redundancy group is configured with a unique host and router name (see Configuring Hostnames).

  2. Each VMR in the HA redundancy group is configured with a unique IP address associated with the Management VRF (see IP Addressing in Hypervisor Environments).
  3. System clocks in both primary and backup routers are synchronized with a Network Time Protocol (NTP) server (see Synchronizing Router Clocks with NTP Servers).
  4. Each VMR in the HA redundancy group is deployed on a different physical host system.
  5. The primary and backup routers are run in identical virtual machines (with the same amount of virtual RAM and same number of virtual processor cores) and use the same size disks for their message spools.
  6. If you are deploying the HA redundancy group in a cloud environment, you must ensure that security rules are created for ports 8300, 8301, 8302 (the three HA configuration synchronization ports), 8741 (the HA mate link port), and 55555 (the SMF port). See the following tables for details:

    Inbound Firewall Rules for VMR 1

    PermissionProtocolSourceDestinationPort
    AllowTCPVMR2 IP
    Monitoring Node IP
    VMR1 IP8300
    AllowTCP and UDPVMR2 IP
    Monitoring Node IP
    VMR1 IP8301
    AllowTCP and UDPVMR2 IP
    Monitoring Node IP
    VMR1 IP8302
    AllowTCP VMR2 IPVMR1 IP8741
    AllowTCPVMR2 IPVMR1 IP55555

    Inbound Firewall Rules for VMR 2

    PermissionProtocolSourceDestinationPort
    AllowTCPVMR1 IP
    Monitoring Node IP
    VMR2 IP8300
    AllowTCP and UDPVMR1 IP
    Monitoring Node IP
    VMR2 IP8301
    AllowTCP and UDPVMR1 IP
    Monitoring Node IP
    VMR2 IP8302
    AllowTCP VMR1 IPVMR2 IP8741
    AllowTCPVMR1 IPVMR2 IP55555

    Inbound Firewall Rules for Monitoring Node

    PermissionProtocolSourceDestinationPort
    AllowUDPVMR1 IP
    VMR2 IP
    Monitoring Node IP8300
    AllowTCP and UDPVMR1 IP
    VMR2 IP
    Monitoring Node IP8301
    AllowTCP and UDPVMR1 IP
    VMR2 IP
    Monitoring Node IP8302

Configuration Parameters

The following table lists parameter values that are required in the configuration procedure.

Parameter Description Example Values
Interface Name The name of the interface the Management VRF uses. To determine the name, enter the show ip vrf management command—it is the name displayed in the Interface column without the “:<#>” suffix. intf0
IP Interface Name The IP interface name is the interface name concatenated with a “:1”.

intf0:1

IP Address

A VMR’s unique IP address/netmask.

To determine the IP address, enter the show ip vrf management command—it is the IP address displayed in the output for the Static V Router.

Primary router: 192.168.132.16/20
Backup router: 192.168.132.39/20
Monitoring node: 192.168.132.3/20
HA redundancy group parameters

The VMR name, IP address or fully qualified domain name (FQDN), and the port number for each node.

Note:  

  • The node name must be the same as the router name.
  • Redundancy requires three ports, which must be accessible by all nodes in a redundancy group. Port 8300 is set by default, and the two subsequent ports are used as required.

Primary router: solace1, connect-via 192.168.132.16

Backup router: solace2, connect-via 192.168.132.39

Monitoring node: solace3, connect-via 192.168.132.3

A node role assigned for each node.

Primary: active-standby-role primary

Backup: active-standby-role standby

Monitoring: Monitoring node doesn’t have the active-standby-role assigned.

Step 2: Configure VMR Redundancy Nodes

To configure an HA redundancy group, you must configure the primary, backup, and monitoring nodes for Direct messaging. Once all the three nodes are configured, they are effectively enabled for Direct Messaging.

To configure the VMR redundancy nodes, perform the following steps in order:

  1. Configure the Primary Router
  2. Configure the Backup Router
  3. Configure the Monitoring Node

Tip:  For a VMR HA configuration, Solace recommends to shutdown message-spool first before shutting down the redundancy.

Configure the Primary Router

To configure the primary router for an HA redundancy group, perform the following steps:

  1. On the router that will act as the primary (solace1 in the examples), shut down the message spool:

    solace1> enable
    solace1# configure
    solace1(configure)# hardware message-spool shutdown
    All message spooling will be stopped.
    Do you want to continue (y/n)? y

  2. Configure the hostlist failover mechanism:

    solace1(configure)# redundancy
    solace1(configure/redundancy)# switchover-mechanism hostlist
    solace1(configure/redundancy)# exit

  3. Configure the HA redundancy group, assign an active-standby-role of primary to this router, and then enable redundancy:

    solace1(configure)# redundancy
    solace1(configure/redundancy)# group
    solace1(configure/redundancy/group)# create node solace1
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace1(configure/redundancy/group/node)# node-type message-routing-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# create node solace2
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace1(configure/redundancy/group/node)# node-type message-routing-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# create node solace3
    solace1(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace1(configure/redundancy/group/node)# node-type monitor-node
    solace1(configure/redundancy/group/node)# exit
    solace1(configure/redundancy/group)# password <password>
    solace2(configure/redundancy/group)# exit
    solace1(configure/redundancy)# active-standby-role primary
    solace1(configure/redundancy)# no shutdown

    Where:

    password is the password used for authentication between nodes in a HA Group. This password must be the same on each node in the HA Group.

Configure the Backup Router

To configure the backup router for an HA redundancy group, perform the following steps:

  1. On the router that will act as the backup (solace2 in the examples), shut down the message spool:
  2. solace2> enable
    solace2# configure
    solace2(configure)# hardware message-spool shutdown
    All message spooling will be stopped
    Do you want to continue (y/n)? y

  3. Configure hostlist as the failover mechanism:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# switchover-mechanism hostlist

  4. Configure the HA redundancy group, assign an active-standby-role of backup to this router, and then enable redundancy:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# group
    solace2(configure/redundancy/group)# create node solace1
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace2(configure/redundancy/group/node)# node-type message-routing-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# create node solace2
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace2(configure/redundancy/group/node)# node-type message-routing-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# create node solace3
    solace2(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace2(configure/redundancy/group/node)# node-type monitor-node
    solace2(configure/redundancy/group/node)# exit
    solace2(configure/redundancy/group)# password <password>
    solace2(configure/redundancy/group)# exit
    solace2(configure/redundancy)# active-standby-role backup
    solace2(configure/redundancy)# no shutdown

    Where:

    password is the password used for authentication between nodes in a HA Group. This password must be the same on each node in the HA Group.

Configure the Monitoring Node

To configure the monitoring node for an HA redundancy group, perform the following steps:

  1. On the router that will act as the monitoring node (solace3 in the examples), reload the default configuration:

    solace3# reload default-config monitoring-node
    This command causes a reload of the system
    Do you want to continue (y/n)? y

    Note:   

    • For VMRs running as Docker images, after reloading the default configuration for the monitoring node, you must start the Docker container manually.

    • To change the IP address of the monitoring node (if required), see IP Addressing in Hypervisor Environments.

  2. Configure the hostlist failover mechanism:

    solace3(configure)# redundancy
    solace3(configure/redundancy)# switchover-mechanism hostlist

  3. Configure the HA redundancy group and enable redundancy.

    Notice that as a monitoring node, this router doesn’t have the active-standby-role assigned.

    solace3(configure)# redundancy
    solace3(configure/redundancy)# group
    solace3(configure/redundancy/group)# create node solace1
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.16
    solace3(configure/redundancy/group/node)# node-type message-routing-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# create node solace2
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.39
    solace3(configure/redundancy/group/node)# node-type message-routing-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# create node solace3
    solace3(configure/redundancy/group/node)# connect-via 192.168.132.3
    solace3(configure/redundancy/group/node)# node-type monitor-node
    solace3(configure/redundancy/group/node)# exit
    solace3(configure/redundancy/group)# password <password>
    solace2(configure/redundancy/group)# exit
    solace3(configure/redundancy)# no shutdown

    Where:

    password is the password used for authentication between nodes in a HA Group. This password must be the same on each node in the HA Group.

Step 3: Set Up Guaranteed Messaging

By default, an HA redundancy group does not have Guaranteed messaging enabled. Guaranteed messaging can only be enabled after primary, backup, and monitoring nodes have redundancy configured. Although Guaranteed messaging is optional, it is recommended because Guaranteed messaging is required for the group to:

  • accept clients that send or receive Guaranteed messages
  • use the recommended Config-Sync functionality

To enable Guaranteed Messaging for the VMR Redundancy Group, perform the following steps:

  1. Enable Guaranteed Messaging
  2. Enable Config-Sync

Enable Guaranteed Messaging

To enable Guaranteed messaging for a VMR HA redundancy group, perform the following steps for the primary and backup routers:

  1. Configure the mate link connect-via parameter on both primary and backup routers.

    The connect-via parameter value for each router must be the IP address or FQDN of its mate. That is, the primary router must point to the static IP address or FQDN of the backup router, and the backup router must point to the static IP address or FQDN of the primary router as shown below.

    Linking with Connect-Via Property

    Guaranteed Messaging Connect-Via Property

    On the primary router, enter the following commands :

    solace1(configure)# redundancy
    solace1(configure/redundancy)# shutdown
    solace1(configure/redundancy)# mate-link connect-via 192.168.132.39
    solace1(configure/redundancy)# no shutdown
    solace1(configure/redundancy)# exit

    On the backup router, enter the following commands:

    solace2(configure)# redundancy
    solace2(configure/redundancy)# shutdown
    solace2(configure/redundancy)# mate-link connect-via 192.168.132.16
    solace2(configure/redundancy)# no shutdown
    solace2(configure/redundancy)# exit

  2. Enable the message spool on both routers.

    On the primary router, enter the following commands:

    solace1(configure)# hardware message-spool
    solace1(configure/hardware/message-spool)# no shutdown
    solace1(configure/hardware/message-spool)# exit
    solace1(configure/hardware)# exit

    On the backup router, enter the following commands:

    solace2(configure)# hardware message-spool
    solace2(configure/hardware/message-spool)# no shutdown
    solace2(configure/hardware/message-spool)# exit
    solace2(configure/hardware)# exit

  3. Start the mate-link service on both routers.

    On the primary router, enter the following commands:

    solace1(configure)# service mate-link
    solace1(configure/service/mate-link)# no shutdown
    solace1(configure/service/mate-link)# exit
    solace1(configure/service)# exit

    On the backup router, enter the following commands:

    solace2(configure)# service mate-link
    solace2(configure/service/mate-link)# no shutdown
    solace2(configure/service/mate-link)# exit
    solace2(configure/service)# exit

  4. To verify that Guaranteed Messaging for the group becomes enabled, use the show redundancy command. The values of ADB Link To Mate, ADB Hello To Mate, and Message Spool Status indicate that the Guaranteed Messaging is enabled.
    • The primary router should have Up and AD-Active values:

      solace1(configure/redundancy)# show redundancy
      Configuration Status     : Enabled
      . . . 
      ADB Link To Mate         : Up
      ADB Hello To Mate        : Up
      . . . 
      Message Spool Status           AD-Active
      . . . 

    • The backup router should have Up and AD-Standby values:

      solace2(configure/redundancy)# show redundancy
      Configuration Status     : Enabled
      . . . 
      ADB Link To Mate         : Up
      ADB Hello To Mate        : Up
      . . . 
      Message Spool Status           AD-Standby
      . . . 

Enable Config-Sync

After a VMR HA redundancy group is configured to support Guaranteed messaging, use Config-Sync to synchronize configurations between primary and backup routers.

Note:  For config-sync to be enabled, make sure that the inbound TCP rule at port 55555 is allowed for the mate router to connect. See Prerequisites for more information.

To enable Config-Sync for the HA group, perform the following steps:

  1. Enable Config-Sync.

    On the primary (solace1) router enter the following commands:

    solace1(configure)# config-sync
    solace1(configure/config-sync)# no shutdown
    solace1(configure/config-sync)# exit
    solace1(configure)# exit

    On the backup (solace2) router enter the following commands:

    solace2(configure)# config-sync
    solace2(configure/config-sync)# no shutdown
    solace1(configure/config-sync)# exit
    solace1(configure)# exit

  2. Assert the primary router’s configuration.

    When enabling the Config-Sync for the first time on an HA pair, you must assert the system-level configuration of the master router (usually it’s the primary router in the pair) over its mate.

    As a result of this operation the Open Status of Config-Sync should change from Down to Up on both primary and backup routers (check it with the show config-sync User EXEC command).

    solace1# admin
    solace1(admin)# config-sync
    solace1(admin/config-sync)# assert-master router
    Processed 1 config-sync tables.

  3. Assert one Message VPN’s configuration.

    If the Message VPNs’ configurations are out of sync, you must manually synchronize a Message VPN configuration between two routers. To do this, you must select one of the message VPNs as a master (it doesn’t have to be on the master or primary router), and assert its configuration over the same Message VPN on the other router.

    solace1(admin/config-sync)# assert-master message-vpn myvpn
    WARNING: This command can temporarily disconnect clients on the
    AD-inactive appliance. As well in-flight messages may not be delivered to AD endpoints on the AD-active appliance if those endpoints are not currently configured the same as they are on this appliance.
    Do you want to continue (y/n)? y
    Processed 1 config-sync tables.

Step 4: Validate Failover

To validate the HA group’s failover operation, perform the following steps:

  1. Manually release activity on the primary router (see Releasing Router Activity), and test that messages are being published and received by the backup router.

    solace1> enable
    solace1# configure
    solace1(configure)# redundancy
    solace1(configure/redundancy)# release-activity

  2. Validate that all the clients successfully reconnect to the backup router after activity is released on the primary router, and test messages continue to be published and received as expected.
  3. Manually take the activity back to the primary router.

    solace1(configure/redundancy)# no release-activity
    solace1(configure/redundancy)# home

  4. Force the backup router to give up activity (see Forcing Backups to Give Up Activity to Primaries).

    solace2> enable
    solace2# admin
    solace2(admin)# redundancy
    solace2(admin/redundancy)# revert-activity

  5. Validate that all the clients reconnect back to the primary router after the activity is released on the backup router, and test that messages continue to be published and received as expected.