Message Spooling

As a PubSub+ event broker receives a Guaranteed message (that is, messages with persistent or non-persistent delivery modes), it processes the message to determine if there are any registered topic subscriptions or queues that match the destination the message was published to. If there is a topic subscription match or a matching queue on the event broker, the message and all the matches are spooled, and then the event broker acknowledges receipt of the message. After acknowledging the message, the event broker attempts to deliver it to all the matching clients and event brokers. As each client and event broker acknowledges receipt of the message, the associated match is deleted from the match list of the message. Once there are no matches left associated with a message, the message itself is deleted from the spool.

If clients are offline or have fallen behind

If one or more of the clients are offline, or have fallen behind, the message is held in the ADB until it can be delivered. If there are too many messages to hold in the ADB’s memory, messages are written to the disk in large blocks. This means only messages for slow or offline clients need to be written to the disk, and they can be written in an efficient manner. If all of the clients acknowledge the message quickly enough, the message doesn't need to be written to disk.

Checks performed before a message is spooled

It's possible for a message to not be spooled because of resource or operating limitations. A variety of checks are performed before a message is spooled. These include:

• Would spooling the message exceed the event broker-wide message spool quota?
• Would spooling the message exceed the Message VPN’s message spool quota?
• Would spooling the message exceed the endpoint’s message spool quota?
• Would spooling the message exceed the endpoint’s maximum permitted message size?
• Would the message exceed the endpoint’s maximum message size?
• Is the destination endpoint shutdown?
• If the message is a low-priority message, would spooling the message exceed the endpoint’s reject low‑priority message limit?

Depending on the reason for why the message was not spooled to the endpoint, either no acknowledgment is returned to the publisher, or a negative acknowledgment (that is, a ‘nack’) is returned, and it's up to the publisher to handle these possibilities. A statistic is then incremented on the event broker.

Message Spool Files

Guaranteed messages are spooled to an event broker service through the use of spool files. These files are stored on a message spool and the spool size you choose should support the number of messages that may need to be stored. If the event broker’s spool files are all in use, it cannot receive any more messages until some spooled messages are acknowledged, which could free some space in the message spool.

If an event broker reaches its maximum spool file usage, negative acknowledgments (that is, 'NACKs') are returned to all publishing clients.Thresholds can be configured so that events are generated when the usage of the message spool gets too high.

For details, refer to Spool Files Thresholds.

Windowed Acknowledgment

A windowed acknowledgment mechanism is used at the transport level between the event broker and individual clients publishing and receiving Guaranteed messages.

A windowed acknowledgment prevents the round-trip acknowledgment time from becoming the limiting factor for message throughput. It does this by allowing a configurable number of messages to be in transit between an event broker and a publishing or subscribing client before an acknowledgment is required. The window size can be configured through a Solace messaging API flow property.

Solace APIs also batch acknowledgments from the application to the event broker. The figure below shows a client application sending multiple acknowledgments for Guaranteed messages, which the API consolidates into a single acknowledgment on the wire that is returned to the event broker. The size of the batch is configured through an API flow property.

With any windowed acknowledgment scheme there is the possibility of failure in the time between a message being received by the client, and the time at which the event broker receives the acknowledgment. A failure at this time requires all non-acknowledged messages in transit to be sent again. Thus, the number of messages redelivered increases and is directly proportional to the combination of window size and acknowledgment threshold.

Cumulative ACK and Acknowledgment Thresholds

Delivered-But-Unacknowledged Messages

There is a hard limit for the number of Guaranteed messages that can be delivered through a consumer flow to a bound client without that client returning an acknowledgment for those messages. On reaching this limit (plus one window size of messages), the event broker stops delivering messages to the client until the client acknowledges messages that have already been delivered.

You can configure the maximum number of delivered‑but‑unacknowledged messages limit for a queue or topic endpoint provisioned on a Message VPN (refer to Message VPN-Level Guaranteed Messaging Configuration).

An event broker has a system-level limit for the maximum number delivered‑but‑unacknowledged messages for all clients at a given time. On reaching this maximum, no more messages are delivered to clients until some clients return acknowledgments back to the event broker, or they are disconnected. The maximum number delivered‑but‑unacknowledged messages supported is dependent on the event broker model.

By default, an event is generated when the number of outstanding messages that have not been acknowledged by their receiving clients reaches 80 percent of the system maximum (the set value), and this is followed by a further event the number of client-unacknowledged messages returns below 60 percent of the system maximum (the clear value). The thresholds for when these events are generated are configurable (refer to Delivered Unacked Thresholds).

Message Expiry

To set a limit on the amount of time that published messages have to be consumed and acknowledged, you can assign Time‑to-Live (TTL) expiry values to them.

• If your application is using the Solace messaging APIs or REST service to publish Guaranteed messages, you can set a Time‑to-Live (TTL) expiry value on each published Guaranteed message to indicate how long the message should be considered valid. A publisher TTL expiration starts when a message is published and counts down as the message passes through the network.
• You can configure a Max Time to Live (TTL) value for a durable endpoint so that received messages are provided with expiration value to limit how long they can remain on that durable endpoint when a Max TTL is used. The Max TTL only applies when a message is on the endpoint.

When TTL values are applied to messages in either or both of these ways, messages that are not consumed and acknowledged before their expiration times are reached are discarded or moved to a Dead Message Queue (DMQ) . If a message has both a publisher-assigned TTL and an endpoint-assigned Max TTL, the event broker will use the minimum of the two TTL values when the message is on the endpoint.

If a Message VPN bridge is used so that published messages that match topic subscriptions can be delivered from a remote Message VPN on one event broker to a local Message VPN on another event broker, the amount of time the message spends on each event broker is counted. That is, the amount of time a message spends on the remote event broker is counted, and its remaining time to live is updated when it is sent to the local Message VPN. For example, with a publisher-provided TTL of eight seconds, if a message spends two seconds on the first event broker, before it reaches the local Message VPN on the second event broker, it will have a TTL of six seconds.

Using TTLs to expire messages that have not been processed quickly can help prevent stale messages from being delivered to consumers. However, it should be noted that monitoring and processing messages using TTLs can affect the system-level limits for Guaranteed message delivery (for more information, refer to Guaranteed Message Queuing Limits).

Maximum Spool Usage

You can set a maximum value for the amount of spool disk space (max-spool-usage) that can be consumed by various objects on the broker. The following tables list the default settings for this value and for the corresponding event thresholds.

System-Level Settings

Broker Type	Max Spool Usage (MB)	Set Threshold (%)	Clear Threshold (%)
Software Event Broker	Default: 1,500 Maximum: 6,000,000	80	60
Appliance (3230/3530)	Default: 60,000 Maximum: 800,000	80	60
Appliance (3260/3560)	Default: 60,000 Maximum: 6,000,000	80	60

Message VPN Settings

Broker Type	Max Spool Usage (MB)	Set Threshold (%)	Clear Threshold (%)
All	Default: 0 Maximum: Same as the maximum for the System-level setting for max-spool-usage	80	60

Settings for Queues / Queue Templates / Topic Endpoints / Topic Endpoint Templates / Client-Provisioned Endpoints

Broker Type	Max Spool Usage (MB)	Set Threshold (%)	Clear Threshold (%)	Set Threshold (MB)	Clear Threshold (MB)
All	Default: 5000 Maximum: Same as the maximum for the System-level setting for max-spool-usage	25	18	1250	900

See the following links for information on setting max-spool-usage values:

• at the system level
• for queues
• for queue templates
• for topic endpoints
• for topic endpoint templates

See the following links for information on setting the event thresholds for message spool usage:

• at the system level
• for queues
• for queue templates
• for topic endpoints
• for topic endpoint templates
• for client-provisioned endpoints