In-Service Replacement of a Failed Appliance Event Broker in a Redundant Pair

This section describes in general terms how to replace a failed Solace Appliance Event Broker in a deployed high-availability (HA) redundant pair. The steps that must be performed are dependent on whether the appliance event broker to be replaced was configured as active or backup and whether the Config‑Sync facility can be used to copy the configuration to the replacement appliance event broker.

ESD/Safety Precautions

To prevent equipment damage from Electrostatic Discharge (ESD), always ensure that the appliance event broker chassis is electrically connected to earth ground. Always follow ESD prevention procedures when removing and replacing power supplies. Use an anti-static wrist strap, or another anti-static device. If no wrist strap or mat is available, ground yourself by touching the metal part of the chassis.

Tool and Equipment Requirements

The following tools and equipment are required:

electrostatic discharge wrist strap
anti-static mat
replacement appliance event broker

Procedure

Complete the following steps to replace a failed appliance event broker in a deployed HA redundant pair. In this example, solace-primary is the active appliance event broker and solace-backup is the appliance event broker that has failed and must be replaced.

If you require further assistance, or if you have any questions regarding this procedure, contact Solace for technical support.

When replacing a failed appliance event broker in a deployed HA redundant pair, do NOT physically connect the HBA links between the replacement appliance event broker and the external disk storage unit, do NOT physically connect the NAB in the replacement appliance event broker to the network, and do NOT physically connect the ADB links between the replacement appliance event broker and paired in-service appliance event broker until explicitly instructed to do so in the steps in the following procedure.

Enter the following Solace Event Broker CLI show command to ensure that solace-primary is the currently active appliance event broker and it is in the correct redundancy state.
```
solace-primary> enable
solace-primary# show redundancy
```

Enter the following commands to ensure that auto-revert is not enabled.

solace-primary> enable
solace-primary# configure
solace-primary(configure)# redundancy
solace-primary(configure/redundancy)# no auto-revert
solace-primary(configure/redundancy)# end
solace-primary#

Install the replacement appliance event broker in the hardware rack, power it up, and connect a management console.
For more information, see Hardware Installation, On / Standby Button, and Connecting a Management Console.
Using the management console, run set-up and configure only the management IP address and default route for the replacement appliance event broker.
For more information, see Setting Up a Basic Configuration and Configuring the Management Interface and Message Backbone Interface on Appliance event brokers.

Enter the following show command to display the HBA Port and Node names for the replacement appliance event broker:

solace-backup> show hardware detail

Example output:

...
Slot 1/3: Host Bus Adapter Blade
  Product #: HBA-0204FC-02-A
  Serial #: M54687
  Model Name: QLE2462
  Model Description: PCI-Express to 4Gb FC, Dual Channel
  Driver Version: 8.01.07-k1
 
  Fibre-Channel 1
    State: Link Up - F_Port (fabric via point-to-point)
    Speed: 2 Gbit
    Port Id: 0x031f00
    Port Name: 0x210000e08b931f25 
    Port Type: NPort (fabric via point-to-point)
    Node Name: 0x200000e08b931f25

Using the displayed HBA Port and Node names from the output, register the HBA ports with the proper storage Logical Unit Number (LUN) on the external disk storage array.
Enter the following show command to ensure that the correct version of Solace software is installed on the replacement appliance event broker.
```
solace-backup> show version
```
If necessary, upgrade or downgrade the version to match that of the version used by the mate appliance event broker. For information on upgrading software on appliance event brokers, see the Solace Appliance Event Broker Release Notes. For information on downgrading appliance event brokers, see Changing Appliance Event Broker Software.
To apply the configuration of the active event broker to the replacement appliance event broker, you can either apply a saved backup configuration file to the replacement mate appliance event broker or use the Config-Sync facility to apply the configuration of the active event broker to the replacement mate appliance event broker:

To apply a backup configuration file to the replacement appliance event broker, see step 9.
To use the Config-Sync facility to assert the active event broker's configuration onto the replacement appliance event broker, see step 10.

To apply a saved configuration file to the replacement appliance event broker, do the following:
1. Perform the procedure outlined in Restoring Configuration Files.
2. Enter the following commands to ensure that auto-revert is not enabled for the replacement appliance event broker:
```
solace-backup> enable
solace-backup# configure
solace-backup(configure)# redundancy
solace-backup(configure/redundancy)# no auto-revert
solace-backup(configure/redundancy)# end
solace-backup#
```
3. Turn off power to the replacement appliance event broker:
```
solace-backup# power-down
```
  The appliance event brokers do not restart automatically after this command is run.
4. Physically connect the HBA links between the replacement appliance event broker and the external disk storage array.
5. Connect the NAB in the replacement appliance event broker to the network.
6. Physically connect the ADB mate links between the replacement appliance event broker and the paired in-service appliance event broker.
7. Power up the replacement appliance event broker:
  See On / Standby Button.
8. Assert the configuration settings of the active appliance event broker onto the replacement appliance event broker. See Asserting Event Broker System Configurations and Asserting Event Brokers VPN Configurations.
9. Verify that the replacement appliance event broker's configuration fully matches the active appliance event broker's configuration.
  You can use the show current-config User EXEC command to obtain the sequence of CLI commands that are required to duplicate the system-level and Message VPN-level configurations used by the active appliance event broker. For more information, see Show the CLI Sequence for the Current Configuration.
To use Config-Sync to automatically synchronize the configurations of the mate event brokers, perform the following steps for the replacement appliance event broker:
1. Configure the Message Backbone (LAG & IP).
  See Configuring the Management Interface and Message Backbone Interface on Appliance event brokers.
2. Configure a redundancy interface.
  See Configuring Redundancy Parameters.
3. Configure its redundancy parameters.
  See Configuring Redundancy Parameters.
4. Configure its message spool.
  See Guaranteed Messaging Configuration.
5. Release activity from the given appliance event broker to its mate:
```
solace-backup(configure/redundancy)# release-activity
```
6. Manually set the configuration parameters that are unique to the appliance event broker and cannot be set through the Config-Sync facility. For a list of Config-Sync ineligible parameters, see Properties That Are Not Synchronized.
7. Enable Config-Sync:
```
solace-backup(configure)# config-sync
solace-backup(configure/config-sync)# no shutdown
```
8. Power down the appliance event broker:
```
solace-backup(configure/config-sync)# end
solace-backup# power-down
```
  The appliance event brokers do not restart automatically after this command is run.
9. Connect the HBA links between the replacement appliance event broker and the external disk storage array.
10. Connect the NAB in the replacement appliance event broker to the network.
11. Connect the ADB links between the replacement appliance event broker and the paired in-service appliance event broker
12. Power up the appliance event broker.
  See On / Standby Button.
13. Assert the configuration settings of the existing event broker onto the replacement event broker. See Asserting Event Broker System Configurations and Asserting Event Brokers VPN Configurations
14. Verify that the configuration is fully in sync. See Viewing Config-Sync Info.
15. Force the appliance event broker to take activity back from its mate:
```
solace-backup(configure/redundancy)# no release-activity
```
16. (Optional) Revert activity.
If necessary, restore the auto-revert redundancy setting.
Solace recommends keeping the auto-revert redundancy configuration disabled. However, if you want to enable auto-revert, re-enable it at this point.
Enter the following commands to verify that the replacement event broker is in the correct redundancy state.
```
solace-backup> enable
solace-backup# show redundancy
```
If it is not in the correct redundancy state, run revert-activity.

Provide feedback