Scaling Up an Existing HA Group

This section shows you the steps to increase the scaling parameters for the primary and backup software event brokers in a high-availability (HA) group using the Solace CLI .

This procedure changes system-wide scaling parameters. Upon completion, you may need to modify other parameters for individual Message VPNs. For example, you may need to modify the maximum number of client connections permitted for a specific Message VPN (Configuring Maximum Connections) or for a specific client profile (see Configuring Max Connections Per Username). For more information, see Configuring Message VPNs.

System scaling parameters can only be increased.

This procedure is service affecting. To change system scaling parameters other than max-subscriptions or the Kafka scaling parameters (Kafka bridges or Kafka broker connections), you must shut down the message backbone and message spool.

The following steps show you how to increase the scaling parameters for event brokers for both the primary and backup:

Step 1: Review System Resource Requirements and HA Status

Step 2: Increase the Value of the Scaling Parameters

Step 1: Review System Resource Requirements and HA Status

Before you increase the value of any scaling parameters of your brokers:

  • Ensure that you can adequately provision your system for the new scaling parameter values. See System Resource Requirements for details about the available tiers and the required resources for each one.
  • Verify that HA is operationally up on all event brokers in the HA group and that the Activity Status for each event broker is correct. You can use the show redundancy command to check operational states, as shown in the following example.

For example, if you have two brokers in an HA pair, where solace-primary is the primary and solace-backup is the backup, the output of the show redundancy command looks similar to the following for the primary broker (note that Redundancy Status is Up, Active-Standby Role is Primary, the Activity Status of the Primary Virtual Router is Local Active, and the Message Spool Status is AD-Active):

solace-primary# show redundancy 

Configuration Status     : Enabled
Redundancy Status        : Up
Operating Mode           : Message Routing Node
Switchover Mechanism     : Hostlist
Auto Revert              : No
Redundancy Mode          : Active/Standby
Active-Standby Role      : Primary
Mate Router Name         : solace-backup
ADB Link To Mate         : Up
ADB Hello To Mate        : Up

                               Primary Virtual Router  Backup Virtual Router
                               ----------------------  ----------------------
Activity Status                Local Active            Shutdown
Routing Interface              intf0:1                 intf0:1
Routing Interface Status       Up                      
VRRP Status                    Initialize              
VRRP Priority                  250                     
Message Spool Status           AD-Active               
Priority Reported By Mate      Standby

The output of the show redundancy command looks similar to the following for the backup broker (note that Redundancy Status is Up, Active-Standby Role is Backup, the Activity Status of the Backup Virtual Router is Mate Active, and the Message Spool Status is AD-Standby):

solace-backup# show redundancy 

Configuration Status     : Enabled
Redundancy Status        : Up
Operating Mode           : Message Routing Node
Switchover Mechanism     : Hostlist
Auto Revert              : No
Redundancy Mode          : Active/Standby
Active-Standby Role      : Backup
Mate Router Name         : solace-primary
ADB Link To Mate         : Up
ADB Hello To Mate        : Up

                               Primary Virtual Router  Backup Virtual Router
                               ----------------------  ----------------------
Activity Status                Shutdown                Mate Active
Routing Interface              intf0:1                 intf0:1
Routing Interface Status                               Up
VRRP Status                                            Initialize
VRRP Priority                                          100
Message Spool Status                                   AD-Standby
Priority Reported By Mate                              Active

Step 2: Increase the Value of the Scaling Parameters

The primary and backup event brokers within an HA group must be configured to use the same system scaling parameters. If this is misconfigured, the HA group will not work properly. You can use the show redundancy command to check operational states.

For replicated sites, the primary and replication event broker sites should have the same resources and maximum number of client connections.

  • Scaling parameters are not automatically synchronized. For further information, refer to Config-Sync Configuration.
  • This procedure changes system-wide scaling parameters. After completing this procedure, you can then modify other parameters for specific Message VPNs (for example, Configuring Maximum Connections) or for clients using a specific client profile (see Configuring Services).

Before proceeding with these steps, ensure that the message spool status of the primary event broker is AD-Active and the message spool status of the backup broker is AD-Standby by running show message-spool or following the example in Step 1: Review System Resource Requirements and HA Status.

In the steps that follow, solace-primary is the primary event broker, and solace-backup is the backup event broker.

In versions 10.9.0 and earlier of the event broker, any change to a single parameter requires an event broker restart. However, with versions 10.9.1 and later, you can now upscale multiple parameters before an event broker restart is required. To increase the value of system scaling parameters for an HA group, click on your event broker version below and follow the steps provided:

  1. On the primary event broker, in the Solace CLI, enter the show system detail command to review the current scaling values. You can also review the memory, cores, and storage requirements that correspond to the requirements output from the system resource calculator (see System Resource Calculator). This command helps you evaluate if additional resources must be added: 
    solace-primary> show system detail

System Uptime: 1d 0h 45m 21s
Last Restart Reason: User request

Scaling:
   Max Connections: 100
   Max Bridges: 25 
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0
   Max Queue Messages: 100M
   Max Subscriptions: 500000

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled

System Resources for Virtual Machine

System Resource                   Available    Required     Units
--------------------------------- ------------ ------------ ----------
CPU
  Cores                           2            2
Memory
---Press any key to continue, or `q' to quit---
  RAM                             5.0          4.0          GiB
Storage Devices
  /dev/dm-0                       9.8 Mi       12.9 Mi      1K-Blocks
    contains: spool-cache
    contains: spool-cache-backup
    contains: config
    contains: diagnostics
    contains: jail
    contains: spool
    contains: var
  /dev/dm-6                       6.0 Mi       1.4 Mi       1K-Blocks
    contains: root
  2. On the backup event broker, shut down the message backbone service and message spool:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool

    solace-backup(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y
solace-backup(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y
  3. On the primary event broker, shut down the message backbone service and message spool:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool.

    solace-primary(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y
solace-primary(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y
  4. On the primary event broker, update the required system scaling parameters.

    You can increase the values of the following system scaling parameters:

    There are two ways to increase scaling parameter values:
    • Non-Interactive CLI mode: Use the scale command followed by the parameter name and value. You can upscale as many parameters as required in one command using this pattern: scale <parameter 1> <value 1> <parameter 2> <value 2> ... <parameter n> <value n>. For example, step 1 shows that the initial value of the client connection limit is 100, and the maximum number of queue messages is 100 million. To increase the connection limit to 1,000 and the queue message limit to 240 million, enter the following commands:
      solace(configure)# system
solace(configure/system)# scaling
solace(configure/system/scaling)# scale max-connections 1000 max-queue-messages 240
This command causes a reload of the system.
Do you want to continue (y/n)? y
Moving ADB messages to disk  : 100%
Backing up ADB config to disk: 100%
Performing database consolidation
    • Interactive CLI mode: Use the scale command without any parameters to display a list of your current scaling parameter values. The CLI then prompts you to enter new values for each parameter in the following order:
      1. connection-limit
      2. max-bridges
      3. max-kafka-bridges
      4. max-kafka-broker-connections
      5. max-messages-in-millions
      6. max-subscriptions
      For each prompt, the current parameter value is displayed in square brackets, and if you do not enter a new value, the value remains the same. For example, step 1 shows that the initial value of the client connection limit is 100, and the maximum number of queue messages is 100 million. To increase the connection limit to 1,000 and the queue message limit to 240 million, enter the following commands:
      solace(configure)# system
solace(configure/system)# scaling
solace(configure/system/scaling)# scale 
Current Scaling:
    Max Connections: 100
    Max Bridges: 25
    Max Kafka Bridges: 0
    Max Kafka Broker Connections: 0
    Max Queue Messages: 100M
    Max Subscriptions: 500000
Enter max connections [100]: 1000

Max bridges should not be greater than max connections (1000)
Enter max bridges [25]:

Max kafka bridges should not be greater than max connections (1000)
Enter max kafka bridges [0]: 

Enter max kafka connections [0]:

Enter max queue messages in millions [100]: 240

Max subscriptions should be greater than max connections (1000)
Enter max subscriptions [500000]:

Updated Scaling:
    Max Connections: 1000
    Max Bridges: 25
    Max Kafka Bridges: 0
    Max Kafka Broker Connections: 0
    Max Queue Messages: 240M
    Max Subscriptions: 500000                                
This command causes a reload of the system.
Do you want to continue (y/n)? y                                    
                                                                        
Moving ADB messages to disk  : 100%
Backing up ADB config to disk: 100%

Performing database consolidation

    5. Changing the value of system scaling parameters causes the system to reload, stopping the container. For container images, you may need to manually restart the container, depending on the restart policy of your container runtime.

    If your HA cluster is running in Kubernetes, the backup and monitoring pods may restart automatically. During this process, the READY status reported by kubectl show pods may be incorrect.

  5. Restart the container if required, then enter the Solace CLI again and confirm the new parameter has been applied. In this example, the maximum number of client connections has been increased to 1,000 and maximum number of queue messages to 240 million.
    solace-primary> show system

System Uptime: 1d 0h 49m 13s
Last Restart Reason: User request

Scaling:
   Max Connections: 1000
   Max Bridges: 25
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0
   Max Queue Messages: 240M
   Max Subscriptions: 500000

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled
  6. In the Solace CLI, enable the message spool and message backbone for the primary event broker:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool.

    solace-primary(configure)# no hardware message-spool shutdown
solace-primary(configure)# no service msg-backbone shutdown
  7. On the primary event broker, verify that the message spool’s Operational Status is AD-Active: 
    solace-primary(configure/service/msg-backbone) show message-spool

Config Status:                            Enabled (Primary)
. . .

Operational Status:                       AD-Active
. . .

                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000      19.0375         19.0375
Number of Messages Currently Spooled            0        1 000           1 000

    If you are increasing parameters other than max-subscriptions or the Kafka scaling parameters (Kafka bridges or Kafka broker connections), the primary message spool status must be AD-Active before continuing with this procedure or message loss will occur.

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), verify that the status on the primary event broker is AD-Standby and on the backup it is AD-Active. When you upscale the backup, the status will change to AD-Standby.

  8. On the backup event broker, update the system scaling parameters to match the settings on the primary broker.

    Changing the value of system scaling parameters causes the system to reload, stopping the container. For container images, you may need to manually restart the container, depending on the restart policy of your container runtime.

  9. Restart the container if required, then enter the Solace CLI again and confirm the new parameter has been applied by running the show system command.

    For example:

    solace-backup> show system

System Uptime: 1d 0h 49m 13s
Last Restart Reason: User request

Scaling:
   Max Bridges: 25
   Max Connections: 1000
   Max Queue Messages: 240M
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0
   Max Subscriptions: 500000

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled
  10. In the Solace CLI, enable the message spool and message backbone service for the backup event broker:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool.

    solace-backup(configure)# no hardware message-spool shutdown
solace-backup(configure)# no service msg-backbone shutdown
  11. On the backup event broker, verify that the message spool’s Operational Status is AD-Standby: 
    solace-backup(configure/service/msg-backbone)# show message-spool
        
Config Status:                            Enabled (Backup)
. . . 

Operational Status:                       AD-Standby
. . .
        
                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000       0.0000          0.0000
Number of Messages Currently Spooled            0            0               0
  12. Verify that the Redundancy Status is Up on all nodes in the HA group:

    On the primary event broker:

    solace-primary> show redundancy

    On the backup event broker:

    solace-backup> show redundancy

    On the monitoring node:

    solace-monitoring> show redundancy
  1. On the primary event broker, in the Solace CLI, enter the show system detail command to review the current scaling values. You can also review the memory, cores, and storage requirements that correspond to the requirements output from the system resource calculator (see System Resource Calculator). This command helps you evaluate if additional resources must be added: 
    solace-primary> show system detail

System Uptime: 1d 0h 45m 21s
Last Restart Reason: User request

Scaling:
   Max Bridges: 25
   Max Connections: 100
   Max Queue Messages: 100M
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0
   Max Subscriptions: 500000

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled

System Resources for Virtual Machine

System Resource                   Available    Required     Units
--------------------------------- ------------ ------------ ----------
CPU
  Cores                           2            2
Memory
---Press any key to continue, or `q' to quit---
  RAM                             5.0          4.0          GiB
Storage Devices
  /dev/dm-0                       9.8 Mi       12.9 Mi      1K-Blocks
    contains: spool-cache
    contains: spool-cache-backup
    contains: config
    contains: diagnostics
    contains: jail
    contains: spool
    contains: var
  /dev/dm-6                       6.0 Mi       1.4 Mi       1K-Blocks
    contains: root
  2. On the backup event broker, shut down the message backbone service and message spool:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool

    solace-backup(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y
solace-backup(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y
  3. On the primary event broker, shut down the message backbone service and message spool:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool.

    solace-primary(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y
solace-primary(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y
  4. On the primary event broker, update the required system scaling parameter.

    You can increase the values of the following system scaling parameters:

    For example, Step 1 shows that the initial value of the client connection limit is 100. To increase the connection limit to 1,000, enter the following commands:

    solace-primary(configure)# system
solace-primary(configure/system)# scaling
solace-primary(configure/system/scaling)# max-connections 1000

    Changing the value of system scaling parameters causes the system to reload, stopping the container. For container images, you may need to manually restart the container, depending on the restart policy of your container runtime.

    5. If your HA cluster is running in Kubernetes, the backup and monitoring pods may restart automatically. During this process, the READY status reported by kubectl show pods may be incorrect.

  5. Restart the container if required, then enter the Solace CLI again and run the show system command to confirm the new parameter has been applied. In this example, the maximum number of client connections has been increased to 1,000.
    solace-primary> show system

System Uptime: 1d 0h 49m 13s
Last Restart Reason: User request

Scaling:
   Max Bridges: 25
   Max Connections: 1000
   Max Queue Messages: 100M
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0
   Max Subscriptions: 500000

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled
  6. Optionally, repeat the preceding two steps to increase another scaling parameter.
  7. In the Solace CLI, enable the message spool and message backbone for the primary event broker:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool.

    solace-primary(configure)# no hardware message-spool shutdown
solace-primary(configure)# no service msg-backbone shutdown
  8. On the primary event broker, verify that the message spool’s Operational Status is AD-Active: 
    solace-primary(configure/service/msg-backbone) show message-spool

Config Status:                            Enabled (Primary)
. . .

Operational Status:                       AD-Active
. . .

                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000      19.0375         19.0375
Number of Messages Currently Spooled            0        1 000           1 000

    If you are increasing parameters other than max-subscriptions or the Kafka scaling parameters (Kafka bridges or Kafka broker connections), the primary message spool status must be AD-Active before continuing with this procedure or message loss will occur.

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), verify that the status on the primary event broker is AD-Standby and on the backup it is AD-Active. When you upscale the backup, the status will change to AD-Standby.

  9. On the backup event broker, update the system scaling parameters to match the settings on the primary broker.

    For example, to increase the connection limit to 1,000, enter the following commands:

    solace-backup(configure)# system
solace-backup(configure/system)# scaling
solace-backup(configure/system/scaling)# max-connections 1000

    Changing the value of system scaling parameters causes the system to reload, stopping the container. For container images, you may need to manually restart the container, depending on the restart policy of your container runtime.

  10. Restart the container if required, then enter the Solace CLI again and confirm the new parameter has been applied by running the show system command.

    For example:

    solace-backup> show system

System Uptime: 1d 0h 49m 13s
Last Restart Reason: User request

Scaling:
   Max Bridges: 25
   Max Connections: 1000
   Max Queue Messages: 100M
   Max Kafka Bridges: 0
   Max Kafka Broker Connections: 0
   Max Subscriptions: 500000

Topic Routing:
   Subscription Exceptions: Enabled
   Subscription Exceptions Defer: Enabled
  11. Optionally, repeat the preceding two steps to increase another scaling parameter to match the settings on the primary event broker.
  12. In the Solace CLI, enable the message spool and message backbone service for the backup event broker:

    If you are increasing max-subscriptions or the Kafka scaling parameters (max-kafka-bridges or max-kafka-broker-connections), you do not need to shut down the msg-backbone or message-spool.

    solace-backup(configure)# no hardware message-spool shutdown
solace-backup(configure)# no service msg-backbone shutdown
  13. On the backup event broker, verify that the message spool’s Operational Status is AD-Standby: 
    solace-backup(configure/service/msg-backbone)# show message-spool
        
Config Status:                            Enabled (Backup)
. . . 

Operational Status:                       AD-Standby
. . .
        
                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000       0.0000          0.0000
Number of Messages Currently Spooled            0            0               0
  14. Verify that the Redundancy Status is Up on all nodes in the HA group:

    On the primary event broker:

    solace-primary> show redundancy

    On the backup event broker:

    solace-backup> show redundancy

    On the monitoring node:

    solace-monitoring> show redundancy