You can increase certain system scaling limits using system scaling parameters. You can specify system scaling parameters using configuration keys when you create a software broker instance (see Setting Scaling Parameters Using Configuration Keys). You can also change system scaling parameters while the software broker is running using the Solace CLI (see Scaling Up an Existing Single Software Event Broker and Scaling Up an Existing HA Group).
For details about the minimum system resources required, see System Resource Requirements.
To see the current values of the system scaling parameters, run the show system command in the Solace CLI, and look at the Scaling section of the output:
solace> show system System Uptime: 1d 0h 45m 21s Last Restart Reason: User request Scaling: Max Bridges: 25 Max Connections: 100 Max Queue Messages: 100M Max Kafka Bridges: 0 Max Kafka Broker Connections: 0 Max Subscriptions: 500000 Topic Routing: Subscription Exceptions: Enabled Subscription Exceptions Defer: Enabled
You can configure the following system scaling parameters:
- Maximum Number of Client Connections
- Maximum Number of Queue Messages
- Maximum Number of VPN Bridges
- Maximum Number of Subscriptions
- Maximum Number of Kafka Bridges
- Maximum Number of Kafka Broker Connections
Increasing a scaling parameters causes an immediate event broker restart.
If you increase the value of these system scaling parameters, the event broker will require additional resources. For more information, see Additional System Resources.
-
Scaling parameters that are decoupled from
max-connectionscan be independently scaled. Independent scaling parameters allow flexibility to scale up deployments without upgrading to larger software event brokers. -
If you don't provide a specific value for an independent scaling parameter, it defaults to a value that is appropriate for your selected value of
max-connections.
Maximum Number of Client Connections
You can increase the number of concurrent client connections supported by the event broker using the max-connections system scaling parameter. By default, the event broker is limited to 100 concurrent client connections, but it can be provisioned for up to 200,000 connections assuming that sufficient system resources are provided.
To scale parameters independent of max-connections, you must upgrade your PubSub+ event broker to version 10.7.1 or later. See Event Broker Releases for release timelines of individual scaling parameters.
| PubSub+ Broker | Default Concurrent Client Connections | Maximum Possible Concurrent Client Connections |
|---|---|---|
| PubSub+ Standard | 100 | 1,000 |
| PubSub+ Enterprise | 100 | 200,000 |
For default maximum client connections values, see maxconnectioncount in Configuration Keys.
The following functionality is not available on software event brokers configured for a maximum of 100 client connections:
To use these features, configure your software event brokers with a higher maximum number of client connections.
For PubSub+ Standard Edition, if you set the max-connections system scaling parameter higher than the maximum supported, the broker will fail to start.
The number of current client connections cannot be changed on a broker configured as a monitoring node in an HA group.
A number of related system limits (for example, the maximum number of client usernames) are automatically scaled to values appropriate for the permitted number of client connections; see the System Limits and Alerts spreadsheet (available from the Solace Products site for Appliance and PubSub+ Enterprise customers) for more information regarding those system limits.
The following types of clients use connections that count against the total number of max-connections:
-
Messaging clients (SMF, MQTT, AMQP, or REST): 1 connection each
-
Internal clients (VPN, VPN Bridge, Kafka Bridge Sender or Receiver, or DMR link per VPN): 1 connection each
Setting the Number of Concurrent Open Files for Container Images
To ensure reliable operation of the event broker, you must also increase the maximum number of concurrent open files per process when the maximum number of client connections is increased. The maximum number of concurrent open files for the container processes is governed by the nofiles resource limit. Most container runtimes support setting rlimit values using the ulimit parameter when creating a container. For example, in Docker Engine and Podman, for 10,000 client connections, add the --ulimit nofile=2448:72992 command line parameter when you create the container.
| Resource Limit | Maximum Client Connections | ||||
|---|---|---|---|---|---|
| 100 | 1,000 | 10,000 | 100,000 | 200,000 | |
| nofiles | 2448:37392 | 2448:40992 | 2448:72992 | 2448:252992 | 2448:452992 |
For ease of use, and if the resources are available, we recommend using 2500:500000 for all instances.
Maximum Number of Queue Messages
You can increase the number of queue messages (references to messages queued for delivery to consumers) supported by the event broker using the max-queue-messages system scaling parameter.
For PubSub+ event broker versions 10.8.1 and later, for new broker instances, the default value for the maximum number of queue messages is governed by the value of the max-connections system scaling parameter:
-
When
max-connectionsis 100, the default value for the maximum number of queue messages is 100 million. -
When
max-connectionsis 1K or higher, the default value for the maximum number of queue messages is 240 million.
| PubSub+ Broker | Maximum Possible Queue Messages |
|---|---|
| PubSub+ Standard | 240,000,000 |
| PubSub+ Enterprise | 3,000,000,000 |
For default maximum queue messages values, see maxqueuemessagecount in Configuration Keys.
Running an event broker with 100M queue messages is not recommended in a production environment as it may result in performance issues.
For PubSub+ Standard Edition, if you set the max-queue-messages system scaling parameter higher than the maximum supported, the broker will fail to start.
The maximum number of queue messages cannot be changed on a broker configured as a monitoring node in an HA group.
Maximum Number of VPN Bridges
You can increase the number of VPN bridges for a single event broker using the max-bridges system scaling parameter. A VPN bridge consumes a client connection. Therefore, don't configure max-bridges values higher than max-connections.
This independent scaling parameter is available for PubSub+ event broker versions 10.7.1 and later.
The value of the maximum number of VPN bridge subscriptions is:
-
100,000 when the
max-bridgesvalue is 5000. -
20,000 when the
max-bridgesvalue is 25 or 500.
| PubSub+ Broker | Maximum Possible VPN Bridges |
|---|---|
| PubSub+ Standard | 500 |
| PubSub+ Enterprise | 5,000 |
For default maximum VPN bridges values, see maxbridgecount in Configuration Keys.
Because max-bridges is dependent on max-connections, if you for example increase max-bridges
from 25 to 5000 and max-connections remains at 100, not all 5000 VPN bridges may be allocated. The Solace CLI shows a warning in such a scenario.
connect-via is limited to 500 outbound VPN bridge connections. This means that in an event mesh, the configuration of 5000 VPN bridges must be done as an inbound connection to the core event brokers.
Maximum Number of Subscriptions
You can increase the number of subscriptions for a single event broker using the max-subscriptions system scaling parameter. This helps to achieve flexibility to deploy different sizes of the event broker within your event mesh without worrying about exceeding the subscription limits of the smaller event brokers in your event mesh . Note that you must allocate additional RAM to the smaller event brokers to accommodate the larger subscription tables. Each client requires at least one subscription. Therefore, don't configure max-connections values higher than max-subscriptions.
This independent scaling parameter is available for PubSub+ event broker versions 10.8.1 and later.
| PubSub+ Broker | Maximum Possible Subscriptions |
|---|---|
| PubSub+ Standard | 500,000 |
| PubSub+ Enterprise | 5,000,000 |
For default maximum subscriptions values, see maxsubscriptioncount in Configuration Keys.
Because max-connections is dependent on max-subscriptions, if your max-connections
is for example at 100 and you increase your max-subscriptions from 50,000 to 500,000, not all 500,000 subscriptions may be allocated. The Solace CLI shows a warning in such a scenario.
For PubSub+ event broker versions earlier than 10.8.1, all event brokers in the same network must have the same value for max-subscriptions; otherwise, subscription resource limits might be exceeded on the event brokers, and message forwarding will not be reliable.
Maximum Number of Kafka Bridges
You can increase the number of Kafka bridges (Kafka senders + Kafka receivers) supported by the event broker using the max-kafka-bridges system scaling parameter. By default, the event broker is limited to 0 bridges, but it can be provisioned for up to 200 bridges assuming that sufficient system resources are provided. In addition, keep in mind that you should not set the max-kafka-bridges value to greater than the max-connections value set for the system.
| PubSub+ Broker | Maximum Possible Kafka Bridges |
|---|---|
| PubSub+ Standard or Enterprise | 200 |
For default maximum subscriptions values, see maxkafkabridgecount in Configuration Keys.
The maximum number of Kafka bridges cannot be changed on a broker configured as a monitoring node in an HA group.
Maximum Number of Kafka Broker Connections
You can increase the number of concurrent Kafka broker connections supported by the event broker using the max-kafka-broker-connections system scaling parameter. By default, the event broker is limited to 0 concurrent Kafka broker connections, but it can be provisioned for up to 10000 connections assuming that sufficient system resources are provided.
You can expect each Kafka sender and receiver to require approximately one connection from this budget for every Kafka broker in the Kafka cluster that they are configured to send messages to or receive messages from. If different senders and receivers connect to the same Kafka cluster these connections are all independent of one another and count separately against this limit. You should, in such cases, use a common sender or receiver with more queue or topic bindings to aid in scaling. The number of bindings does not affect the number of required connections.
When deciding on the maximum number of concurrent Kafka broker connections your deployment will support, you must ensure that enough process IDs (PIDs) are available to your platform. Kafka bridging requires that 1000 + the value you configure for max-kafka-broker-connections PIDs are available. For more information about setting per Node and per Pod PIDs limits in Kubernetes, see Process ID Limits And Reservations.
| PubSub+ Broker | Maximum Possible Concurrent Kafka Broker Connections |
|---|---|
| PubSub+ Standard or Enterprise | 10,000 |
For default maximum subscriptions values, see maxkafkabrokerconnectioncount in Configuration Keys.
The maximum number of Kafka broker connections cannot be changed on a broker configured as a monitoring node in an HA group.
If the event broker exceeds the value configured for the maximum number of Kafka broker connections, Kafka senders and receivers on the event broker may become operationally down with failure reasons such as Too many other broker connections in use or Too many broker connections required. To prevent this, you can monitor the current number of Kafka broker connections using the show kafka and show message-vpn <vpn-name> kafka CLI commands (see Monitoring Kafka Bridges). In addition, the event broker generates SYSTEM_KAFKA_BROKER_CONNECTIONS_HIGH and VPN_KAFKA_BROKER_CONNECTIONS_HIGH events when it approaches the maximum value. For more information, see Solace PubSub+ Syslog Events.
Multiple Broker Considerations
If you have multiple event brokers in an event mesh, you must be aware of the following considerations when increasing the scaling parameters:
-
Except for HA group setups, the values of the system scaling parameters can be arbitrarily different on each software event broker in the event mesh.
-
For
max-subscriptionsspecifically, because the allocation of SMRP can be unpredictable, if you increase to a higher subscription limit on one event broker, you must update the other event brokers in the event mesh to the same value as soon as possible.
Additional System Resources
The resource requirements for the minimum (default) values of the system scaling parameters are listed in System Resource Requirements.
If you increase the value of any system scaling parameters, the event broker will require additional resources. To see how the system resource requirements change with higher values of system scaling parameters, use the System Resource Calculator.