Message Delivery Resources
This section describes the egress per-client priority queues on PubSub+ event brokers and the commands that you can use to configure them.
Client Egress Queue Structure Overview
For a published message to be delivered to a client, it passes through the queuing structure shown in the Egress Client Queue Hierarchy figure below.
The message first passes through one of the five per-client priority data queues. A scheduler then selects it and places it into a single, per-client Transmission Control Protocol (TCP) transmit queue.
For Direct messages, the Class of Service (COS) value the publishing client application sets for the message determines which per-client priority queue each message is enqueued to:
- COS 1 (C-1)—the message goes to the Direct 1 (D-1) queue (the lowest priority)
- COS 2 (C-2)—the message goes to the Direct 2 (D-2) queue (medium priority)
- COS 3 (C-3)—the message goes to the Direct 3 (D-3) queue (the high priority)
All Guaranteed messages are enqueued to the Guaranteed 1 (G-1) queue. A Guaranteed message also has an assigned COS value, however, this value determines the message’s discard eligibility when its destination endpoint is congested. COS 1 indicates that the message is low priority; COS 3 indicates that the message is high priority. (COS 2 is a reserved Solace value; currently a message with a COS of 2 is treated as low priority.)
The act of enqueuing the message triggers the TCP stack to evaluate if the connection can send more data from that queue. If the TCP stack determines that it is acceptable to send data on the TCP connection, then data from the message is copied to a per-port transmit queue. After the message is delivered, the data remains on the TCP transmit queue until it is acknowledged by the client through a TCP ACK message.
The per-client priority queues have a configurable maximum depth that is measured in work units of 2,048 bytes. When these queues become full, the older messages currently queued are discarded (oldest first), replaced by the newer incoming messages.
Once a message discard occurs due to the queue being full, the receiving client is notified of it through a discard indication published with the message immediately following the discarded messages. The data plane statistic for Egress Congestion Discards is also incremented.
The per-client TCP transmit queues hold messages that are either waiting for delivery out of the event broker, or that have already been sent, but are waiting for TCP acknowledgment.
Data Buffer Management
An event broker protects against a small number of clients consuming too many internal buffers. This protection applies to both ingress and egress traffic between event brokers and clients, and is most typically used on:
- Publishing clients that are sending data to the event broker at a rate that is disproportionately consuming memory resources.
- Slow subscribers. That is, clients that consistently fail to consume their messages at the offered message rate. To determine how slow a subscriber is, the event broker measures the number of seconds in the last minute when it had data to send but could not, typically because the client’s TCP window was closed. An additional penalty is applied to the slowness measurement for TCP problems such as retransmissions and missed TCP Keepalives. Slowness is expressed as a percentage.
To view any slow subscribers on an event broker, enter the show client * slow-subscriber User EXEC command.
To view a subscriber’s blocked percentage, enter the show client <name> connections wide User EXEC command.
Ingress Data Buffer Management
Event brokers ensure fair processing of messages from publishing clients. A publishing client cannot consume a disproportionate amount of ingress resources at the expense of other publishing clients.
CLIENT_CLIENT_MESSAGE_TOO_BIG event is raised.Egress Data Buffer Management
An event broker uses an internal gauge to monitor its egress buffer usage called the NAB Buffer Load Factor, whereby egress data buffer usage is registered as a percentage. When this gauge exceeds 100%, slow subscribers can be automatically disconnected to reduce the egress data buffer load.
If the egress data buffer usage on the event broker is getting high, a NAB Buffer Load Factor event based on the current percentage value of this gauge is generated before any slow subscribers are disconnected from the event broker.
To configure threshold values for the NAB Buffer Load Factor event, refer to Configuring System Memory Usage Event Thresholds.
Configuring Egress Per-Client Priority Queues
To avoid data loss or service interruption, always contact Solace for technical support before you attempt to configure client queues.
To configure the egress per-client priority queues, enter the following commands:
solace(configure)# client-profile <name> message-vpn <vpn-name>
solace(configure/client-profile)# queue <type>
Where:
<name> is the name of the client profile. A client profile name can contain up to 32 alphanumeric characters (case-sensitive). The “_” character is also permitted.
<vpn-name> is the name of the Message VPN the client profile belongs to.
<type> is the queue type to be configured, that is, one of the following client priority queue parameters (in order of priority):
C-1(Control 1)D-3(Direct 3)D-2(Direct 2)D-1(Direct 1)G-1(Guaranteed 1)
You can configure the following egress queue parameters for the given queue:
Configuring Max Egress Queue Depths
To configure the maximum depth of the given queue for clients, enter the following command:
solace(configure/client-profile/queue)# max-depth <depth>
Where:
<depth> is the integer value representing the number of work units for the client priority queues. The valid range is 2 to 262144 for client priority queues. The default is 20,000 work units for the client priority queues. Changing this value does not affect messages already successfully placed on the queue.
The no version of this command, no max-depth, resets the queue depth for the specified queue to the default of 20000.
- For Web Messaging clients using the message eliding feature, Solace recommends setting a maximum queue depth that is sufficient to hold one message for each unique topic that a client may receive.
- For all other Web Messaging clients, Solace recommends setting a maximum queue depth of 100. This recommendation is based on the assumption that the event broker is loaded with concurrent client connections near the maximum permitted value.
Configuring Egress Queue Minimum Message Bursts
To configure the number of messages that are always allowed entry into the specified priority queue, regardless of the max-depth value, enter the following command:
solace(configure/client-profile/queue)# min-msg-burst <depth>
Where:
<depth> is the integer value representing the queue burst depth in messages. The valid range is 0 to 262144. The default value is 4 for queues D1-3 and 255 for the G-1 queue.
- The no version of this command (no min-msg-burst) resets the queue burst depth in messages to the default value.
- For the G-1 queue, the minimum message burst value should at least equal the consumer flow window size used by client applications and/or Message VPN bridge clients. If the minimum message burst value is too low, Guaranteed messages will be transmitted at a very reduced rate.
Monitoring Egress Per-Client Priority Queues
You can use the show commands to validate and monitor an event broker’s message queue configurations and status.
Showing Client Profile Detail
To view the message queue configuration for a particular client profile or all client profiles on the event broker, enter the following command:
solace> show client-profile <name> [message-vpn <vpn-name>] [detail]
Where:
<name> is the name of the client profile. Entering no name displays all client profiles configured on the event broker.
<vpn-name>is the full name of the Message VPN that the client profile belongs to, or part of the Message VPN name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all Message VPNs.
Example
solace> show client-profile solprofile detail
Profile Name : solprofile
Message VPN: dotnet_vpn
G-Guaranteed, D-Direct, C-Control)
Queue Max Depths
G-1 : 20000 work units
D-1 : 20000 work units
D-2 : 20000 work units
D-3 : 20000 work units
C-1 : 20000 work units
Priority Queue Min Burst
G-1 : 255 messages
D-1 : 4 messages
D-2 : 4 messages
D-3 : 4 messages
C-1 : 4 messages
Message Spool
Guaranteed Message Send : deny
Guaranteed Message Receive : deny
Guaranteed Endpoint Create : deny
Transacted Sessions : deny
Max Transacted Sessions : 10
Max Endpoints per client-username : 16000
Max Egress Flows : 16000
Max Ingress Flows : 16000
Copy-from queue :
Copy-from Topic Endpoint :
TCP
MSS : 1460
Initial Window : 2
Maximum Window : 256 KB
Keepalive
Count : 5
Idle : 3 seconds
Interval : 1 seconds
Eliding
Enabled : No
Max Topics : 256
Delay : 0 milliseconds
Web Transport
Inactive Timeout : 30 seconds
Max Web Payload : 1000000 bytes
Replication
Allow Clients When Standby : No
Max # Connections per client-username : 6000
Service SMF : 9000
Web Transport : 9000
Max # Subscriptions : 5000000
Allow Bridge Connections : No
Event Threshold Set Value clear Value
---------------------------------- ---------------- ----------------
Subscriptions (#subs) 80%(4000000) 60%(3000000)
Connections per client-username 80%(4800) 60%(3600)
Service SMF 80%(7200) 60%(5400)
Service WEB 80%(7200) 60%(5400)
Provisioned endpt spool usage (%) 80 60
Transacted sessions (#) 80%(8) 60%(6)
Endpoints per client-username (#) 80%(12800) 60%(9600)
Egress flows (#) 80%(12800) 60%(9600)
Ingress flows (#) 80%(12800) 60%(9600)
Legend:
User Type - CU (client-username)
UB (unidirectional bridge [remote message-vpn])
RB (replication bridge)
AG (authorization group)
User Name # Active
Type Client
---- ---------------------------------------------------------------- --------
CU default 0
The following table explains the displayed egress queue parameters.
| Parameter | Description |
|---|---|
|
Queue Max Depths |
Each of the egress per-client priority queues has an associated maximum depth. The depths are measured in work units( a work unit represents 2048 bytes of a message). The formula to convert a message size to number of work units is: NumWorkUnits = CEILING(message.length/2048). When a new message is placed on the priority queue, its depth in work units is checked against the queue's configured maximum depth. Queues can temporarily exceed their configured maximum depths. However, when these queues become full, the older messages currently queued are discarded (oldest first), replaced by the newer incoming messages, and the dataplane statistic for Egress Congestion Discards is incremented. Once a message discard occurs due to the queue being full, the receiving client is notified by a discard indication that is published with the message immediately following the discarded message(s). |
|
Priority Queue Min Burst |
The configured minimum burst length tolerance specifies the number of messages per queue that are always allowed entry into the queue, regardless of its current configured maximum depth. It is used to ensure that messages are not lost when bursts of very large messages are received on a egress client priority queue. For example, in a case where there are three messages currently on a queue, and the minimum burst length tolerance of the queue is set to 4, then one more message is allowed to be placed on the queue, regardless of whether the three current messages are large enough to fill the queue past its maximum allowed depth. Thus, the configured minimum burst length tolerance of 4 allows the queue to temporarily exceed its configured maximum depth and absorb a burst of large messages that would otherwise exceed the number of allowed work units. |
Showing Client Stats Queue
To view the message queue statistics for clients, enter the following command:
solace> show client <name> stats queue
Where:
<name> is the full name of the client, or part of the client name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all clients.
Example
solace> show client solace1 stats queue
Client: solace1 Type: Primary Client Profile: default ACL Profile: default Subscriptions: 2 Eliding Enabled: No Eliding Topics: Current: 0 High Water Mark: 0 Deliver To One: Local Priority: 1 Network Priority: 1 Client Id: 151 Message VPN: v1 Uptime: 0d 0h 2m 52s Slow Subscriber: No Client Username: cache User: 'whani' Computer: 'dev184' Process ID: 11035 Description: Software Version: 5.1.0.223 Software Date: Oct 19 2010 11:47:18 Platform: Linux26-i386_opt - C SDK One Shot Events: Large Message: not raised Message Too Big: not raised Parse Error: not raised Max Eliding Topics: not raised Listed from lowest to highest priority: (G-Guaranteed, D-Direct, C-Control) *************************** Client Transmit Queues *************************** Queue | Length | Length | HWMark | Max | Discards | Delivered (Prio) | (Msgs) | (Work) | (Work) | (Work) | (Msgs) | (Msgs) -------+---------+---------+----------+----------+----------+----------------- G-1 | 0 | 0 | 0 | 20000 | 0 | 0 D-1 | 0 | 0 | 0 | 20000 | 0 | 0 D-2 | 0 | 0 | 0 | 20000 | 0 | 0 D-3 | 0 | 0 | 0 | 20000 | 0 | 0 C-1 | 0 | 0 | 1 | 20000 | 0 | 3
The following table explains the statistics displayed for the egress queues.
| Column Name | Description |
|---|---|
|
Queue (Prio) |
For client TCP transmit queues, lists the queue type, and priority (from low to high): G-1 (Guaranteed 1); D-1 (Direct 1); D-2 (Direct 2); D-3 (Direct 3); C-1 (Control 1). |
|
Length (Msgs) |
The number of messages currently waiting in the queue. |
|
Length (Work) |
The number of work units currently in the queue. |
|
HWMark (Work) |
The maximum depth in work units that the queue has reached since the last time the statistics were cleared. |
|
Max (Work) |
The number of work units that can be placed in the queue before messages are discarded. |
|
Discards (Msgs) |
The number of messages that were discarded from the queue. |
|
Delivered (Msgs) |
The number of messages that were delivered from the queue. |
Showing CSPF Neighbor Stats Queues
To view the message queue statistics for neighbors of an event broker, enter the following command:
solace> show cspf neighbor <physical-message broker-name> stats queues
Where:
<physical-message broker-name> is the full physical event broker name of the neighbor, or part of the physical event broker name of the neighbor with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all neighbors.
Example
solace> show cspf neighbor 1.2.3.4 stats queues
Neighbor: 1.2.3.4
Connect Via: 1.2.3.4:55555
Control Port: unspecified
******************* Control Queues - Control Card ******************
| Length | HWMark
Queue | (Msgs) | Msg)
----------------+---------+---------
CSPF | 0 | 0 |
SMRP | 0 | 0 |
Listed from lowest to highest priority: (G-Guaranteed, D-Direct, C-Control)
****************** Data Queues - Network Acceleration Blade ******************
Queue | Length | Length | HWMark | Max | Discards | Delivered
(Prio) | (Msgs) | (Work) | (Work) | (Work) | Msgs) | (Msgs)
-------+---------+---------+----------+----------+----------+-----------------
Conn-1
G-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-2 | 0 | 0 | 0 | 20000 | 0 | 0
D-3 | 0 | 0 | 0 | 20000 | 0 | 0
Conn-2
G-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-2 | 0 | 0 | 0 | 20000 | 0 | 0
D-3 | 0 | 0 | 0 | 20000 | 0 | 0
Conn-3
G-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-2 | 0 | 0 | 0 | 20000 | 0 | 0
D-3 | 0 | 0 | 0 | 20000 | 0 | 0
Conn-4
G-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-1 | 0 | 0 | 0 | 20000 | 0 | 0
D-2 | 0 | 0 | 0 | 20000 | 0 | 0
D-3 | 0 | 0 | 0 | 20000 | 0 | 0
The following table explains the statistics displayed for the queues.
| Column Name | Description |
|---|---|
|
Queue (Prio) |
For the four TCP data connections comprising the neighbor link (Conn-1, Conn-2, Conn-3, Conn-4), lists the queue type, and priority (from low to high): G-1 (Guaranteed 1); D-1 (Direct 1); D-2 (Direct 2); D-3 (Direct 3). |
|
Length (Msgs) |
The number of messages currently waiting in the queue. |
|
Length (Work) |
The number of work units currently in the queue. |
|
HWMark (Work) |
The maximum depth in work units that the queue has reached since the last time the statistics were cleared. |
|
Max (Work) |
The number of work units that can be placed in the queue before messages are discarded. |
|
Discards (Msgs) |
The number of messages that were discarded from the queue. |
|
Delivered (Msgs) |
The number of messages that were delivered from the queue. |