Setting Scaling Parameters for an HA Group

This section shows you the steps to increase the scaling parameters for the primary and backup software event brokers in a high-availability (HA) group using the Solace CLI .

This procedure changes system-wide scaling parameters. Upon completion, you may need to modify other parameters for individual Message VPNs. For example, you may need to modify the maximum number of client connections permitted for a specific Message VPN (Configuring Maximum Connections) or for a specific client profile (see Configuring Max Connections Per Username). For more information, see Configuring Message VPNs.

System scaling parameters can only be increased.

This procedure is service affecting. To change system scaling parameters other than the Kafka scaling parameters, you must shut down the message backbone and message spool.

Step 1: Review System Resource Requirements and HA Status

Before you increase the value of any scaling parameters of your brokers:

Ensure that you can adequately provision your system for the new scaling parameter values. See System Resource Requirements for details about the available tiers and the required resources for each one.
Verify that HA is operationally up on all event brokers in the HA group and that the Activity Status for each event broker is correct. You can use the show redundancy command to check operational states.

For example, if you have two brokers in an HA pair, where solace_01 is the primary and solace_02 is the backup, the output of the show redundancy command looks similar to the following for the primary broker:

solace_01# show redundancy 

Configuration Status     : Enabled
Redundancy Status        : Up
Operating Mode           : Message Routing Node
Switchover Mechanism     : Hostlist
Auto Revert              : No
Redundancy Mode          : Active/Standby
Active-Standby Role      : Primary
Mate Router Name         : solace_02
ADB Link To Mate         : Up
ADB Hello To Mate        : Up

                               Primary Virtual Router  Backup Virtual Router
                               ----------------------  ----------------------
Activity Status                Local Active            Shutdown
Routing Interface              intf0:1                 intf0:1
Routing Interface Status       Up                      
VRRP Status                    Initialize              
VRRP Priority                  250                     
Message Spool Status           AD-Active               
Priority Reported By Mate      Standby

The output of the show redundancy command looks similar to the following for the backup broker:

solace_02# show redundancy 

Configuration Status     : Enabled
Redundancy Status        : Up
Operating Mode           : Message Routing Node
Switchover Mechanism     : Hostlist
Auto Revert              : No
Redundancy Mode          : Active/Standby
Active-Standby Role      : Backup
Mate Router Name         : solace_01
ADB Link To Mate         : Up
ADB Hello To Mate        : Up

                               Primary Virtual Router  Backup Virtual Router
                               ----------------------  ----------------------
Activity Status                Shutdown                Mate Active
Routing Interface              intf0:1                 intf0:1
Routing Interface Status                               Up
VRRP Status                                            Initialize
VRRP Priority                                          100
Message Spool Status                                   AD-Standby
Priority Reported By Mate                              Active

Step 2: Increase the Value of the Scaling Parameter(s)

The primary and backup event brokers within an HA group must be configured to use the same system scaling parameters. If this is misconfigured, the HA group will not work properly. You can use the show redundancy command to check operational states.

For replicated sites, the primary and replication event broker sites should have the same resources and maximum number of client connections.

Scaling parameters are not automatically synchronized. For further information, refer to Config-Sync Configuration.
This procedure changes system-wide scaling parameters. After completing this procedure, you can then modify other parameters for specific Message VPNs (for example, Configuring Maximum Connections) or for clients using a specific client profile (see Configuring Services).

In the steps that follow, solace_01 is the primary event broker, and solace_02 is the backup event broker.

To increase the value of a system scaling parameter for an HA group, do the following:

On the primary event broker, in the Solace CLI, enter the show system command to review the current scaling values:

solace_01> show system

System Uptime: 1d 0h 45m 21s
Last Restart Reason: User request

Scaling:
   Max Bridges: 25
   Max Connections: 100
   Max Queue Messages: 100M
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled

On the backup event broker, shut down the message backbone service and message spool:

If you are increasing the Kafka scaling parameters (Kafka bridges or Kafka broker connections) you do not need to shut down the msg-backbone or message-spool.

solace_02(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y
solace_02(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y

On the primary event broker, shut down the message backbone service and message spool:

If you are increasing the Kafka scaling parameters (Kafka bridges or Kafka broker connections) you do not need to shut down the msg-backbone or message-spool.

solace_01(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y
solace_01(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y

On the primary event broker, update the required system scaling parameter.
You can increase the values of the following system scaling parameters:
- maximum number of client connections (using the max-connections <connection-limit> command)
- maximum number of queue messages (using the max-queue-messages <max-messages-in-millions> command)
- maximum number of VPN bridges (using the max-bridges <max-bridges> command)
- maximum number of Kafka bridges (using the max-kafka-bridges <max-kafka-bridges> command)
- maximum number of Kafka broker connections (using the max-kafka-broker-connections <max-kafka-broker-connections> command)
For example, Step 1 shows that the initial value of the client connection limit is 100. To increase the connection limit to 1,000, enter the following commands:
```
solace_01(configure)# system
solace_01(configure/system)# scaling
solace_01(configure/system/scaling)# max-connections 1000
```
Changing the value of system scaling parameters causes the system to reload, stopping the container. In machine images, the container automatically restarts. For container images, you may need to manually restart the container, depending on the restart policy of your container runtime.

Restart the container if required, then enter the CLI again and run the show system command to confirm the new parameter has been applied. In this example, the maximum number of client connections has been increased to 1,000.

solace_01> show system

System Uptime: 1d 0h 49m 13s
Last Restart Reason: User request

Scaling:
   Max Bridges: 25
   Max Connections: 1000
   Max Queue Messages: 100M
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled

Optionally, repeat the preceding two steps to increase another scaling parameter.
In the Solace CLI, enable the message spool and message backbone for the primary event broker:
If you are increasing the Kafka scaling parameters (Kafka bridges or Kafka broker connections) you do not need to enable the msg-backbone or message-spool.
```
solace_01(configure)# no hardware message-spool shutdown
solace_01(configure)# no service msg-backbone shutdown
```
On the primary event broker, verify that the message spool’s Operational Status is AD-Active:
```
solace_01(configure/service/msg-backbone) show message-spool
```
The primary message spool status must be AD-Active before continuing with this procedure or message loss will occur.
If you are increasing the Kafka scaling parameters (Kafka bridges or Kafka broker connections) verify that the status on the primary event broker is AD-Standby and on the backup it is AD-Active. When you upscale the backup, the status will change to AD-Standby.
On the backup event broker, update the system scaling parameter to match the settings on the primary broker.
Changing the value of system scaling parameters causes the system to reload, stopping the container. In machine images, the container automatically restarts. For container images, you may need to manually restart the container, depending on the restart policy of your container runtime.
Restart the container if required, then enter the CLI again and confirm the new parameter has been applied by running the show system command.
Optionally, repeat the preceding two steps to increase another scaling parameter to match the settings on the primary event broker.
In Solace CLI, enable the message spool and message backbone service for the backup event broker:
If you are increasing the Kafka scaling parameters (Kafka bridges or Kafka broker connections) you do not need to enable the msg-backbone or message-spool.
```
solace_02(configure)# no hardware message-spool shutdown
solace_02(configure)# no service msg-backbone shutdown
```
On the backup event broker, verify that the message spool’s Operational Status is AD-Standby:
```
solace_02(configure/service/msg-backbone)# show message-spool
```
Verify that the Redundancy Status is Up on all nodes in the HA group:
On the primary event broker:
```
solace_01> show redundancy
```
On the backup event broker:
```
solace_02> show redundancy
```
On the monitoring node:
```
solace-monitoring> show redundancy
```

Provide feedback