Message Spooling

As a Solace router receives Guaranteed messages (that is, messages with persistent or non-persistent delivery modes), it processes that message to determine if there are any registered topic subscriptions or queues that match the destination the message was published to. If there is a topic subscription match or a matching queue on the router, the message and all the matches are spooled and then the router acknowledges receipt of the message. After acknowledging the message, the router attempts to deliver it to all the matching clients/routers. As each client/router acknowledges receipt of the message, the associated match is deleted from the match list of the message. Once there are no matches left associated with a message, the message itself is deleted from the spool (that is, it is discarded).

If one or more of the clients are offline or have fallen behind, the message is held in the ADB until the message can be delivered. If there are too many messages to hold in the ADB’s memory, messages are written to the disk in large blocks. This means only messages for slow/offline clients need to be written to the disk and they can be written in an efficient manner.

Note:  If all of the clients acknowledge the message quickly enough, the message does not need to be written to disk.

It is possible for a message to not be spooled because of resource or operating limitations. A variety of checks are performed before a message is spooled. These include:

  • Would spooling the message exceed the router-wide message spool quota?
  • Would spooling the message exceed the Message VPN’s message spool quota?
  • Would spooling the message exceed the endpoint’s message spool quota?
  • Would spooling the message exceed the endpoint’s maximum permitted message size?
  • Would the message exceed the endpoint’s maximum message size?
  • Is the destination endpoint shutdown?
  • If the message is a low-priority message, would spooling the message exceed the endpoint’s reject low‑priority message limit?

Depending on the reason for why the message was not spooled to the endpoint, either no acknowledgment is returned to the publisher or a negative acknowledgment (that is, a ‘nack’) is returned, and it is up to the publisher to handle these possibilities. A statistic is then incremented on the router.

Note:   

  • If a subscription is deleted when a message is spooled for that subscription, the message will still be delivered.
  • If an endpoint is deleted (for example, through the Solace CLI) while a message is spooled to that endpoint, the message will not be delivered.

Spool Files

Guaranteed messages are spooled to a Solace router through the use of spool files that can each hold approximately 8 MB worth of messages. A router can support one million or more spool files—the maximum number of message spool files available depends on the Solace router model. If the router’s spool file usage reaches this limit, it cannot receive any more messages until some spooled messages are acknowledged, which could free some spool files.

If a router reaches its maximum spool file usage, negative acknowledgments (that is, ‘nacks’) are returned to all publishing clients. However, spool file thresholds can be configured so that events are generated when the system-level message spool usage gets too high but is not exceeded, or when it gets abnormally low. Refer to System-Level Message Spool Configuration.

By default, an event is generated when more than 80% or less than 60% of spool files are in use.

Windowed Acknowledgment

A windowed acknowledgment mechanism is used at the transport level between the router and individual clients publishing and receiving Guaranteed messages.

A windowed acknowledgment prevents the round-trip acknowledgment time from becoming the limiting factor for message throughput. It does this by allowing a configurable number of messages to be in transit between a Solace router and a publishing or subscribing client before an acknowledgment is required. The window size can be configured through a Solace messaging API flow property.

Solace APIs also batch acknowledgments from the application to the router. The figure below shows a client application sending multiple acknowledgments for Guaranteed messages, which the API consolidate into a single acknowledgment on the wire that is returned to the router. The size of the batch is configured through an API flow property.

Note:  With any windowed acknowledgment scheme there is the possibility of failure in the time between a message being received by the client, and the time at which the router receives the acknowledgment. A failure at this time requires all non-acknowledged messages in transit to be sent again. Thus, the number of messages redelivered increases and is directly proportional to the combination of window size and acknowledgment threshold.

Cumulative ACK and Acknowledgment Thresholds

Delivered-But-Unacknowledged Messages

There is a hard limit for the number of Guaranteed messages that can be delivered through a consumer flow to a bound client without that client returning an acknowledgment for those messages. On reaching this limit (plus one window size of messages), the Solace router stops delivering messages to the client until the client acknowledges messages that have already been delivered.

Note:  You can configure the maximum number of delivered‑but‑unacknowledged messages limit for a queue or topic endpoint provisioned on a Message VPN (refer to Message VPN-Level Guaranteed Messaging Configuration).

A Solace router has a system-level limit for the maximum number delivered‑but‑unacknowledged messages for all clients at a given time. On reaching this maximum, no more messages are delivered to clients until some clients return acknowledgments back to the router, or they are disconnected. The maximum number delivered‑but‑unacknowledged messages supported is dependent on the Solace router model.

By default, an event is generated when the number of outstanding messages that have not been acknowledged by their receiving clients reaches 80 percent of the system maximum (the set value), and this is followed by a further event the number of client-unacknowledged messages returns below 60 percent of the system maximum (the clear value). The thresholds for when these events are generated are configurable (refer to Delivered Unacked Thresholds).

Message Expiry

To set a limit on the amount of time that published messages have to be consumed and an acknowledgment of receipt is not returned to the router, you can assign Time‑to-Live (TTL) expiry values to the messages.

  • If your application is using the Solace messaging APIs or REST service to published Guaranteed messages, you can set a Time‑to-Live (TTL) expiry value on each published Guaranteed message to indicate how long the message should be considered valid. A publisher TTL expiration starts when a message is published and counts down as the message passes through the network.
  • You can configure a Max Time to Live (TTL) value for a durable endpoint so that received messages are provided with expiration value to limit how long they can remain on that durable endpoint when a Max TTL is used. The Max TTL only applies when a message is on the endpoint.

When TTL values are applied to messages in either or both of these ways, messages that are not consumed and acknowledged before their expiration times are reached are discarded or moved to a Dead Message Queue (DMQ) . If a message has both a publisher-assigned TTL and an endpoint-assigned Max TTL, the router will use the minimum of the two TTL values when the message is on the endpoint.

Note:  If a Message VPN bridge is used so that published messages that match topic subscriptions can be delivered from a remote Message VPN on one router to a local Message VPN on another router, the amount of time the message spends on each router is counted. That is, the amount of time a message spends on the remote router is counted, and its remaining time to live is updated when it is sent to the local Message VPN. For example, with a publisher-provided TTL of eight seconds, if a message spends two seconds on the first router, before it reaches the local Message VPN on the second router, it will have a TTL of six seconds.

Using TTLs to expire messages that have not been processed quickly can help prevent stale messages from being delivered to consumers. However, it should be noted that monitoring and processing messages using TTLs can affect the system-level limits for Guaranteed message delivery (for more information, refer to Guaranteed Message Queuing Limits).