Replacing Nodes in Software Event Broker HA Redundancy Groups

You can replace any of the nodes in a SolacePubSub+ software event broker high-availability (HA) redundancy group. Before you begin this procedure, ensure that all the event brokers in your HA redundancy group have the same broker version.

The following procedures explain how to replace an existing node with a newly configured one:

Replacing Primary Software Event Brokers
Replacing Backup Software Event Brokers
Replacing Monitoring Nodes

When you are replacing a node in a software event broker HA redundancy group, if the new node uses a different IP address, fully-qualified domain name (FQDN), or router name than the node it is replacing, client applications will lose connectivity. As a result, the client applications must be reconfigured to use the new IP address, FQDN, or router name.

The following procedures cause a service interruption. Service interruption can be prevented if you have configured replication for your event broker and failover to your disaster recovery site before starting the following procedures. For more information, see Data Center Replication for Disaster Recovery.

Replacing Primary Software Event Brokers

To replace the primary software event broker in a HA group, perform the following steps using the Solace CLI:

The examples within this procedure use the following names for the event brokers:

solace1 to represent the primary event broker you are replacing
solace1-new to represent the new primary event broker you are replacing solace1 with
solace2 to represent the backup event broker
solace3 to represent the monitoring node

Shut down the primary event broker so that clients can no longer access it:

If you are running a container, stop the container using the appropriate command for your container runtime. For example, with Docker Engine, you can use the following stop command:
```
sudo docker stop [OPTIONS] CONTAINER [CONTAINER...]
```

On the event broker (solace1-new) that will act as the new primary messaging node, shut down the message spool:

solace1-new> enable
solace1-new# configure
solace1-new(configure)# hardware message-spool shutdown
All message spooling will be stopped
Do you want to continue (y/n)? y

Shut down the message backbone service:

solace1-new(configure)# service
solace1-new(configure/service)# msg-backbone
solace1-new(configure/service/msg-backbone)# shutdown
All clients will be disconnected
Do you want to continue (y/n)? y

Configure any required TLS/SSL service and trusted CA certificates. For more information, see TLS/SSL Service Connections and Managing Server Certificates.
Configure the failover mechanism:

solace1-new(configure)# redundancy
solace1-new(configure/redundancy)# switchover-mechanism hostlist

Configure the HA redundancy group, assign an active-standby-role of primary to this event broker, and then enable redundancy:

solace1-new(configure/redundancy)# group
solace1-new(configure/redundancy/group)# create node solace1-new
solace1-new(configure/redundancy/group/node)# connect-via 192.168.132.116
solace1-new(configure/redundancy/group/node)# node-type message-routing-node
solace1-new(configure/redundancy/group/node)# exit
solace1-new(configure/redundancy/group)# create node solace2
solace1-new(configure/redundancy/group/node)# connect-via 192.168.132.39
solace1-new(configure/redundancy/group/node)# node-type message-routing-node
solace1-new(configure/redundancy/group/node)# exit
solace1-new(configure/redundancy/group)# create node solace3
solace1-new(configure/redundancy/group/node)# connect-via 192.168.132.3
solace1-new(configure/redundancy/group/node)# node-type monitor-node
solace1-new(configure/redundancy/group/node)# exit
solace1-new(configure/redundancy)# authentication
solace1-new(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
solace1-new(configure/redundancy/group)# exit
solace1-new(configure/redundancy)# active-standby-role primary
solace1-new(configure/redundancy)# no shutdown

Where:

<pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base64). It's used to provide authentication between nodes in an HA Group, and must be the same on each node.

On the backup event broker (solace2), shut down the message spool:

solace2> enable
solace2# configure
solace2(configure)# hardware message-spool shutdown
All message spooling will be stopped
Do you want to continue (y/n)? y

If the primary event broker has a different IP address, FQDN, or router name than the node it's replacing, the redundancy group information must be updated on both the backup and monitoring nodes. However, if the primary will use the same IP address or FQDN and router name as the node it has replaced, then go to step 9.

Shut down redundancy on both event brokers in the HA (solace2 is the backup event broker and the monitoring node (solace3).

On the backup event broker (solace2:

solace2> enable
solace2# configure
solace2(configure)# redundancy
solace2(configure/redundancy)# shutdown

On the monitoring node (solace3):

solace3> enable
solace3# configure
solace3(configure)# redundancy
solace3(configure/redundancy)# shutdown

Remove the old node from the redundancy group on both event brokers.

On the backup event broker:

solace2(configure/redundancy)# group
solace2(configure/redundancy/group)# no node solace1

On the monitoring node:

solace3(configure/redundancy)# group
solace3(configure/redundancy/group)# no node solace1

Create a new group node with the new router name and IP address or FQDN on both event brokers.

On the backup event broker:

solace2(configure/redundancy/group)# create node solace1-new
solace2(configure/redundancy/group/node)# connect-via 192.168.132.116
solace2(configure/redundancy/group/node)# node-type message-routing-node
solace2(configure/redundancy/group/node)# exit

On the monitoring node:

solace3(configure/redundancy/group)# create node solace1-new
solace3(configure/redundancy/group/node)# connect-via 192.168.132.116
solace3(configure/redundancy/group/node)# node-type message-routing-node
solace3(configure/redundancy/group/node)# exit

Enable redundancy on the backup and monitoring node.
On the backup event broker:
```
solace2(configure/redundancy)# no shutdown
```
On the monitoring node:
```
solace3(configure/redundancy)# no shutdown
```
Reload the backup node for the change to take effect:
```
solace2(configure)# reload
```

Enable the message spool on the backup event broker (solace2):

solace2(configure)# no hardware message-spool shutdown
solace2(configure)# show message-spool

Confirm Redundancy Status is Up on all nodes in the HA group.
On the primary event broker:
```
solace1-new> show redundancy
```
On the backup event broker:
```
solace2> show redundancy
```
On the monitoring node:
```
solace3> show redundancy
```

On the primary event broker, perform the release-activity operation:

solace1-new> enable
solace1-new# configure
solace1-new(configure)# redundancy
solace1-new(configure/redundancy)# release-activity

Enable the mate link on both the primary and backup event brokers.

On the primary event broker:

solace1-new(configure)# service mate-link
solace1-new(configure/service/mate-link)# no shutdown

On the backup event broker:

solace2(configure)# service mate-link
solace2(configure/service/mate-link)# no shutdown

Enable the message backbone service:

solace1-new> enable
solace1-new# configure
solace1-new(configure)# service
solace1-new(configure/service)# msg-backbone
solace1-new(configure/service/msg-backbone)# no shutdown
solace1-new(configure/service/msg-backbone)# exit
solace1-new(configure/service)# exit

Enable the message spool on the new primary event broker and verify that the message spool’s Operational Status is AD-Active:
```
solace1-new(configure)# no hardware message-spool shutdown
solace1-new> show message-spool
```

Enable Config-Sync.

Set up Config-Sync without TLS:

solace1-new> enable
solace1-new# configure
solace1-new(configure)# config-sync
solace1-new(configure/config-sync)# no shutdown

Set up Config-Sync over TLS:

solace1-new> enable
solace1-new# configure
solace1-new(configure)# config-sync
solace1-new(configure/config-sync)# ssl
solace1-new(configure/config-sync)# no shutdown

Assert solace2 as the Config-Sync leader:

solace2(admin/config-sync)# assert-leader router
Processed 1 config-sync tables.
solace2(admin/config-sync)# assert-leader message-vpn *
WARNING: This command can temporarily disconnect clients on the
AD-inactive appliance. Clients connected to this appliance will not be affected.
Do you want to continue (y/n)? y

Verify that the Operational Status is Up for Config-Sync on both the primary and backup event brokers.
On the primary event broker:
```
solace1-new> show config-sync
```
On the backup event broker:
```
solace2> show config-sync
```

On the primary event broker, run no release-activity to resume activity:

solace1-new> enable
solace1-new# configure
solace1-new(configure)# redundancy
solace1-new(configure/redundancy)# no release-activity

Verify that the Redundancy Status is in the Up status on all nodes in the HA group.
On the primary event broker:
```
solace1-new> show redundancy
```
On the backup event broker:
```
solace2> show redundancy
```
On the monitoring node:
```
solace3> show redundancy
```

If the new primary event broker is using a new router name, IP address, or FQDN, all existing clients’ connection information must be reconfigured to use this new information. For example, for clients using the PubSub+ Messaging APIs, the host property used to establish client sessions must be modified (see Session Properties Required to Establish a Connection).

Replacing Backup Software Event Brokers

To replace the backup event broker in a software event broker HA group, perform the following steps using the Solace CLI:

The examples within this procedure use the following names for the event brokers:

solace1 to represent the primary event broker you are replacing
solace2 to represent the backup event broker
solace2-new to represent the new backup event broker you are replacing solace2 with
solace3 to represent the monitoring node

Shut down the backup event broker so that clients can no longer access it:

If you are running a container, stop the container using the appropriate command for your container runtime. For example, with Docker Engine, you can use the following stop command:
```
sudo docker stop [OPTIONS] CONTAINER [CONTAINER...]
```

On the event broker that will act as the new backup (solace2-new), shut down the message spool.

solace2-new> enable
solace2-new # configure
solace2-new(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y

Shut down the message backbone service:

solace2-new(configure)# service
solace2-new(configure/service)# msg-backbone
solace2-new(configure/service/msg-backbone)# shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y

Configure any required TLS/SSL service and trusted CA certificates. For more information, see TLS/SSL Service Connections and Managing Server Certificates.
Configure the host list failover mechanism:

solace2-new(configure)# redundancy
solace2-new(configure/redundancy)# switchover-mechanism hostlist

Configure the HA redundancy group, assign an active-standby-role of backup to this event broker, and then enable redundancy:

solace2-new(configure/redundancy)# group
solace2-new(configure/redundancy/group)# create node solace1
solace2-new(configure/redundancy/group/node)# connect-via 192.168.132.16
solace2-new(configure/redundancy/group/node)# node-type message-routing-node
solace2-new(configure/redundancy/group/node)# exit
solace2-new(configure/redundancy/group)# create node solace2-new
solace2-new(configure/redundancy/group/node)# connect-via 192.168.132.139
solace2-new(configure/redundancy/group/node)# node-type message-routing-node
solace2-new(configure/redundancy/group/node)# exit
solace2-new(configure/redundancy/group)# create node solace3
solace2-new(configure/redundancy/group/node)# connect-via 192.168.132.3
solace2-new(configure/redundancy/group/node)# node-type monitor-node
solace2-new(configure/redundancy/group/node)# exit
solace2-new(configure/redundancy)# authentication
solace2-new(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
solace2-new(configure/redundancy/group)# exit
solace2-new(configure/redundancy)# active-standby-role backup
solace2-new(configure/redundancy)# no shutdown

Where:

<pre-shared-key> is 44 to 344 characters (which translates into 32 to 256 bytes of binary data encoded in base 64). It's used to provide authentication between nodes in a HA Group, and must be the same on each node.

On the primary event broker (solace1), shut down the message spool:

solace1> enable
solace1# configure
solace1(configure)# hardware message-spool shutdown
All message spooling will be stopped
Do you want to continue (y/n)? y

If the backup event broker (solace2-new) has a different IP address, FQDN, or router name than the node it is replacing, the redundancy group information must be updated on both the primary and monitoring nodes. However, if the backup will use the same IP address or FQDN and router name as the node it has replaced, then go to step 8.

Shut down redundancy on both event brokers.
To avoid message loss, verify that you have stopped the old backup event broker (step 1) and disabled the message backbone service on the new backup event broker (step 3).
On the primary event broker solace1):
```
solace1> enable
solace1# configure
solace1(configure)# redundancy
solace1(configure/redundancy)# shutdown
```
On the monitoring node (solace3):
```
solace3> enable
solace3# configure
solace3(configure)# redundancy
solace3(configure/redundancy)# shutdown
```

Remove the old node from the redundancy group on both the primary event broker and the monitoring node.

On the primary event broker:

solace1(configure/redundancy)# group
solace1(configure/redundancy/group)# no node solace2

On the monitoring node:

solace3(configure/redundancy)# group
solace3(configure/redundancy/group)# no node solace2

Create the new group node with the new router name and IP address or FQDN on both event brokers.

On the primary event broker (solace1):

solace1(configure/redundancy/group)# create node solace2-new
solace1(configure/redundancy/group/node)# connect-via 192.168.132.139
solace1(configure/redundancy/group/node)# node-type message-routing-node
solace1(configure/redundancy/group/node)# exit

On the monitoring node (solace3):

solace3(configure/redundancy/group)# create node solace2-new
solace3(configure/redundancy/group/node)# connect-via 192.168.132.139
solace3(configure/redundancy/group/node)# node-type message-routing-node
solace3(configure/redundancy/group/node)# exit

Enable redundancy on the primary and monitoring nodes.
On the primary event broker:
```
solace1(configure/redundancy)# no shutdown
```
On the monitoring node:
```
solace3(configure/redundancy)# no shutdown
```
Reload the primary node for the change to take effect:
```
solace1(configure/redundancy)# reload
```

Enable the message spool on the primary event broker (solace1):

solace1(configure)# no hardware message-spool shutdown
solace1(configure)# show message-spool

Confirm Redundancy Status is Up on all nodes in the HA group.
On the primary event broker (solace1):
```
solace1> show redundancy
```
On the backup event broker:
```
solace2> show redundancy
```
On the monitoring node:
```
solace3> show redundancy
```

On the backup event broker (solace2-new), release-activity:

solace2-new> enable
solace2-new# configure
solace2-new(configure)# redundancy
solace2-new(configure/redundancy)# release-activity

Enable the mate link on both the primary event broker and the monitoring node.

On the primary event broker:

solace1(configure)# service mate-link
solace1(configure/service/mate-link)# no shutdown

On the backup node:

solace2-new(configure)# service mate-link
solace2-new(configure/service/mate-link)# no shutdown

Enable message backbone service:

solace2-new> enable
solace2-new# configure
solace2-new(configure)# service
solace2-new(configure/service)# msg-backbone
solace2-new(configure/service/msg-backbone)# no shutdown
solace2-new(configure/service/msg-backbone)# exit
solace2-new(configure/service)# exit

Enable the message spool and verify the message-spool Operational Status is AD-Standby:

solace2-new(configure)# no hardware message-spool shutdown
solace2-new> show message-spool

Enable Config-Sync.

Set up Config-Sync without TLS:

solace2-new> enable
solace2-new# configure
solace2-new(configure)# config-sync
solace2-new(configure/config-sync)# no shutdown

Set up Config-Sync over TLS:

solace2-new> enable
solace2-new# configure
solace2-new(configure)# config-sync
solace2-new(configure/config-sync)# ssl
solace2-new(configure/config-sync)# no shutdown

Assert the primary event broker as the Config-Sync leader:

solace1(admin/config-sync)# assert-leader router
Processed 1 config-sync tables.
solace1(admin/config-sync)# assert-leader message-vpn *
WARNING: This command can temporarily disconnect clients on the AD-inactive appliance.
Clients connected to this appliance will not be affected.
Do you want to continue (y/n)? y

Verify that the Config-Sync OperStatus is in the Up status on both the primary and backup event brokers.
On the primary event broker:
```
solace1> show config-sync
```
On the backup event broker:
```
solace2-new> show config-sync
```

On the backup event broker, take back activity:

solace2-new> enable
solace2-new# configure
solace2-new(configure)# redundancy
solace2-new(configure/redundancy)# no release-activity

Verify that the Redundancy Status is in the Up status on all nodes in the HA group.
On the primary event broker:
```
solace1> show redundancy
```
On the backup event broker:
```
solace2-new> show redundancy
```
On the monitoring node:
```
solace3> show redundancy
```

If the new backup event broker is using a new router name, IP address, or FQDN, all existing clients’ connection information must be reconfigured to use this new information. For example, for clients using the PubSub+ Messaging APIs, the host property used to establish client sessions must be modified. For more information, see Session Properties Required to Establish a Connection.

Replacing Monitoring Nodes

To replace the monitoring node for a software event broker HA redundancy group, perform the following steps using the Solace CLI:

The examples within this procedure use the following names for the event brokers:

solace1 to represent the primary event broker you are replacing
solace2 to represent the backup event broker
solace3 to represent the monitoring node
solace3-new to represent the new monitoring node you are replacing solace3 with

Shut down the monitoring node (solace3) so that the primary and backup event brokers can no longer access it:

If you are running a container, stop the container using the appropriate command for your container runtime. For example, with Docker Engine, you can use the following stop command:
```
sudo docker stop [OPTIONS] CONTAINER [CONTAINER...]
```

On the event broker that will act as the new monitoring node (solace3-new in this example), reload the default configuration:

solace3-new# reload default-config monitoring-node
This command causes a reload of the system
Do you want to continue (y/n)? y

Configure the failover mechanism:

solace3-new(configure)# redundancy
solace3-new(configure/redundancy)# switchover-mechanism hostlist

Configure the HA redundancy group and enable redundancy on replacement monitoring node:

Notice that as a monitoring node, this event broker doesn't have the active-standby-role role assigned to it.

solace3-new(configure/redundancy)# group
solace3-new(configure/redundancy/group)# create node solace1
solace3-new(configure/redundancy/group/node)# connect-via 192.168.132.16
solace3-new(configure/redundancy/group/node)# node-type message-routing-node
solace3-new(configure/redundancy/group/node)# exit
solace3-new(configure/redundancy/group)# create node solace2
solace3-new(configure/redundancy/group/node)# connect-via 192.168.132.39
solace3-new(configure/redundancy/group/node)# node-type message-routing-node
solace3-new(configure/redundancy/group/node)# exit
solace3-new(configure/redundancy/group)# create node solace3-new
solace3-new(configure/redundancy/group/node)# connect-via 192.168.132.113
solace3-new(configure/redundancy/group/node)# node-type monitor-node
solace3-new(configure/redundancy/group/node)# exit
solace3-new(configure/redundancy)# authentication
solace3-new(configure/redundancy/authentication)# pre-shared-key key <pre-shared-key>
solace3-new(configure/redundancy/group)# exit
solace3-new(configure/redundancy)# no shutdown

Where

If the new monitoring node has a different IP address, FQDN, or router name than the node it is replacing, the redundancy group information must be updated on both the primary and backup event brokers.

Shut down redundancy on both event brokers.

On the primary event broker:

solace1> enable
solace1# configure
solace1(configure)# redundancy
solace1(configure/redundancy)# shutdown

On the backup event broker:

solace2> enable
solace2# configure
solace2(configure)# redundancy
solace2(configure/redundancy)# shutdown

Remove the old node from the redundancy group on both event brokers.

On the primary event broker:

solace1(configure/redundancy)# group
solace1(configure/redundancy/group)# no node solace3

On the backup event broker:

solace2(configure/redundancy)# group
solace2(configure/redundancy/group)# no node solace3

Create the new group node with the new router name and IP address or FQDN on both event brokers.

On the primary event broker:

solace1(configure/redundancy/group)# create node solace3-new
solace1(configure/redundancy/group/node)# connect-via 192.168.132.113
solace1(configure/redundancy/group/node)# node-type monitor-node
solace1(configure/redundancy/group/node)# exit

On the backup event broker:

solace2(configure/redundancy/group)# create node solace3-new
solace2(configure/redundancy/group/node)# connect-via 192.168.132.113
solace2(configure/redundancy/group/node)# node-type monitor-node
solace2(configure/redundancy/group/node)# exit

Enable redundancy on both event brokers:
On the primary event broker:
```
solace1(configure/redundancy)# no shutdown
```
On the backup event broker:
```
solace2(configure/redundancy)# no shutdown
```

Verify that the Redundancy Status is in the Up status on all nodes in the HA group.
On the primary event broker:
```
solace1> show redundancy
```
On the backup event broker:
```
solace2> show redundancy
```
On the monitoring node:
```
solace3-new> show redundancy
```

Provide feedback