Managing Message Delivery Resources

This section describes the egress per-client priority queues on Solace routers and the commands that you can use to configure them.

Client Egress Queue Structure Overview

For a published message to be delivered to a client, it passes through the queuing structure shown in the Egress Client Queue Hierarchy figure below.

The message first passes through one of the five per-client priority data queues. A scheduler then selects it and places it into a single, per-client Transmission Control Protocol (TCP) transmit queue.

For Direct messages, the Class of Service (COS) value the publishing client application sets for the message determines which per-client priority queue each message is enqueued to:

  • COS 1—the message goes to the Direct 1 queue (the lowest priority)
  • COS 2—the message goes to the Direct 2 queue (medium priority)
  • COS3—the message goes to the Direct 3 queue (the high priority).

All Guaranteed messages are enqueued to the Guaranteed 1 queue. A Guaranteed message also has an assigned COS value, however, this value determines the message’s discard eligibility when its destination endpoint is congested. COS 1 indicates that the message is low priority; COS 3 indicates that the message is high priority. (COS 2 is a reserved Solace value; currently a message with a COS of 2 is treated as low priority.)

The act of enqueuing the message triggers the TCP stack to evaluate if the connection can send more data from that queue. If the TCP stack determines that it is acceptable to send data on the TCP connection, then data from the message is copied to a per-port transmit queue. After the message is delivered, the data remains on the TCP transmit queue until it is acknowledged by the client through a TCP ACK message.

Egress Client Queue Hierarchy

The per-client priority queues have a configurable maximum depth that is measured in work units of 2,048 bytes. When these queues become full, the older messages currently queued are discarded (oldest first), replaced by the newer incoming messages.

Once a message discard occurs due to the queue being full, the receiving client is notified of it through a discard indication published with the message immediately following the discarded messages. The dataplane statistic for Egress Congestion Discards is also incremented.

The per-client TCP transmit queues hold messages that are either waiting for delivery out of the Solace router, or that have already been sent, but are waiting for TCP acknowledgment.

Data Buffer Management

A Solace router protects against a small number of clients consuming too many internal buffers. This protection applies to both ingress and egress traffic between Solace routers and clients, and is most typically used on:

  • Publishing clients that are sending data to the router at a rate that is disproportionately consuming memory resources.
  • Slow subscribers. That is, clients that consistently fail to consume their messages at the offered message rate, which causes their egress queues to fill up. (Clients can also be considered slow subscribers if they fail to respond to TCP Keepalives or the router must send timed TCP retransmissions to the clients.)

To view any slow subscribers on a Solace router, enter the show client * slow-subscriber User EXEC command.

Ingress Data Buffer Management

If a Solace router detects that a backlog of at least 24k buffers has built up in the its message processing stage, and there is received data ready for processing from more than one client or neighbor, then the router gives higher service priority to clients or neighbors that have been recently sending data to the router at a lower‑than‑average rate and lower service priority to clients or neighbors that have been recently sending data to the router at a higher-than-average rate.

If the router detects that a backlog of at least 48k buffers has built up in the router’s message processing stage, then the router momentarily defers service of all ingress TCP data, to limit the ingress data rate to one which it can service.

In either case, when a router stops servicing ingress TCP data on some or all TCP connections, the peer TCP client limits its sending rate and, if necessary, applies back pressure to the sending client application.

Egress Data Buffer Management

A Solace router uses an internal gauge to monitor its egress buffer usage called the NAB Buffer Load Factor, whereby egress data buffer usage is registered as a percentage. When this gauge exceeds 100%, slow subscribers can be automatically disconnected to reduce the egress data buffer load.

If the egress data buffer usage on the router is getting high, a NAB Buffer Load Factor event based on the current percentage value of this gauge is generated before any slow subscribers are disconnected from the router.

To configure threshold values for the NAB Buffer Load Factor event, refer to Configuring System Memory Usage Event Thresholds.

Configuring Egress Per-Client Priority Queues

ALERT! To avoid data loss or service interruption, always contact Solace for technical support before you attempt to configure client queues.

To configure the egress per-client priority queues, enter the following CONFIG commands:

solace(configure)# client-profile <name> message-vpn <vpn-name>

solace(configure/client-profile)# queue <type>

Where:

<name> is the name of the client profile. A client profile name can contain up to 32 alphanumeric characters (case-sensitive). The “_” character is also permitted.

<vpn-name> is the name of the Message VPN the client profile belongs to.

<type> is the queue type to be configured, that is, one of the following client priority queue parameters (in order of priority):

  • C-1 (Control 1)
  • D-3 (Direct 3)
  • D-2 (Direct 2)
  • D-1 (Direct 1)
  • G-1 (Guaranteed 1)

Entering the queue <type> Client Profile CONFIG command moves the CLI to a queue configuration mode for the Client Profile VPN CONFIG level, from which you can configure the following egress queue parameters for the specified queue:

Configuring Max Egress Queue Depths

To configure the maximum depth of the given queue for clients, enter the following CONFIG command:

solace(configure/client-profile/queue)# max-depth <depth>

Where:

<depth> is the integer value representing the number of work units for the client priority queues. The valid range is 50 to 262144 for client priority queues. The default is 20,000 work units for the client priority queues. Changing this value does not affect messages already successfully placed on the queue.

Note:   

  • The no version of this command (no max-depth) resets the queue depth for the specified queue to the default of 20000.
  • For Web Messaging clients using the message eliding feature, Solace recommends setting a maximum queue depth that is sufficient to hold one message for each unique topic that a client may receive.
  • For all other Web Messaging clients, Solace recommends setting a maximum queue depth of 100. This recommendation is based on the assumption that the router is loaded with concurrent client connections near the maximum permitted value.

Configuring Egress Queue Minimum Message Bursts

To configure the minimum number of messages that must be on a client priority queue before the queue’s depth is checked against the maximum depth setting (thereby allowing the queue to absorb a burst of large messages that exceeds the number of allowed work units), enter the following CONFIG command:

solace(configure/client-profile/queue)# min-msg-burst <depth>

Where:

<depth> is the integer value representing the queue burst depth in messages. The valid range is 0 to 262144. The default value is 4 for queues D1-3 and 255 for the G-1 queue.

Note:   

  • The no version of this command (no min-msg-burst) resets the queue burst depth in messages to the default value.
  • For the G-1 queue, the minimum message burst value should at least equal the consumer flow window size used by client applications and/or Message VPN bridge clients. If the minimum message burst value is too low, Guaranteed messages will be transmitted at a very reduced rate.

Monitoring Egress Per-Client Priority Queues

You can use the show commands to validate and monitor a router's message queue configurations and status:

Showing Client Profile Detail

To view the message queue configuration for a particular client profile or all client profiles on the Solace router, enter the following User EXEC command:

solace> show client-profile <name> [message-vpn <vpn-name>] [detail]

Where:

<name> is the name of the client profile. Entering no name displays all client profiles configured on the router.

<vpn-name>is the full name of the Message VPN that the client profile belongs to, or part of the Message VPN name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all Message VPNs.

Showing Client Stats Queue

To view the message queue statistics for clients, enter the following User EXEC command:

solace> show client <name> stats queue

Where:

<name> is the full name of the client, or part of the client name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all clients.

Showing CSPF Neighbor Stats Queues

To view the message queue statistics for neighbors of a Solace router, enter the following User EXEC command:

solace> show cspf neighbor <physical-router-name> stats queues

Where:

<physical-router-name> is the full physical router name of the neighbor, or part of the physical router name of the neighbor with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all neighbors.