As a Solace PubSub+ event broker receives a Guaranteed message (that is, messages with persistent or non-persistent delivery modes), it processes the message to determine if there are any registered topic subscriptions or queues that match the destination the message was published to. If there is a topic subscription match or a matching queue on the event broker, the message and all the matches are spooled, and then the event broker acknowledges receipt of the message. After acknowledging the message, the event broker attempts to deliver it to all the matching clients and event brokers. As each client and event broker acknowledges receipt of the message, the associated match is deleted from the match list of the message. Once there are no matches left associated with a message, the message itself is deleted from the spool.
If clients are offline or have fallen behind
If one or more of the clients are offline, or have fallen behind, the message is held in the ADB until it can be delivered. If there are too many messages to hold in the ADB’s memory, messages are written to the disk in large blocks. This means only messages for slow or offline clients need to be written to the disk, and they can be written in an efficient manner. If all of the clients acknowledge the message quickly enough, the message doesn't need to be written to disk.
Checks performed before a message is spooled
It's possible for a message to not be spooled because of resource or operating limitations. A variety of checks are performed before a message is spooled. These include:
- Would spooling the message exceed the event broker-wide message spool quota?
- Would spooling the message exceed the Message VPN’s message spool quota?
- Would spooling the message exceed the endpoint’s message spool quota?
- Would spooling the message exceed the endpoint’s maximum permitted message size?
- Would the message exceed the endpoint’s maximum message size?
- Is the destination endpoint shutdown?
- If the message is a low-priority message, would spooling the message exceed the endpoint’s reject low‑priority message limit?
Depending on the reason for why the message was not spooled to the endpoint, either no acknowledgment is returned to the publisher, or a negative acknowledgment (that is, a ‘nack’) is returned, and it's up to the publisher to handle these possibilities. A statistic is then incremented on the event broker.
- If a subscription is deleted when a message is spooled for that subscription, the message will still be delivered.
- If an endpoint is deleted (for example, through the Solace CLI) while a message is spooled to that endpoint, the message will not be delivered.
Guaranteed messages are spooled to an event broker through the use of spool files that can each hold approximately 8 MB worth of messages. An event broker can support one million or more spool files—the maximum number of message spool files available depends on the Solace PubSub+ model. If the event broker’s spool file usage reaches this limit, it cannot receive any more messages until some spooled messages are acknowledged, which could free some spool files.
If an event broker reaches its maximum spool file usage, negative acknowledgments (that is, ‘nacks’) are returned to all publishing clients. However, spool file thresholds can be configured so that events are generated when the system-level message spool usage gets too high but is not exceeded, or when it gets abnormally low. Refer to System-Level Message Spool Configuration.
By default, an event is generated when more than 80% or less than 60% of spool files are in use.
A windowed acknowledgment mechanism is used at the transport level between the event broker and individual clients publishing and receiving Guaranteed messages.
A windowed acknowledgment prevents the round-trip acknowledgment time from becoming the limiting factor for message throughput. It does this by allowing a configurable number of messages to be in transit between an event broker and a publishing or subscribing client before an acknowledgment is required. The window size can be configured through a Solace messaging API flow property.
Solace APIs also batch acknowledgments from the application to the event broker. The figure below shows a client application sending multiple acknowledgments for Guaranteed messages, which the API consolidate into a single acknowledgment on the wire that is returned to the event broker. The size of the batch is configured through an API flow property.
: With any windowed acknowledgment scheme there is the possibility of failure in the time between a message being received by the client, and the time at which the event broker receives the acknowledgment. A failure at this time requires all non-acknowledged messages in transit to be sent again. Thus, the number of messages redelivered increases and is directly proportional to the combination of window size and acknowledgment threshold.
Cumulative ACK and Acknowledgment Thresholds
There is a hard limit for the number of Guaranteed messages that can be delivered through a consumer flow to a bound client without that client returning an acknowledgment for those messages. On reaching this limit (plus one window size of messages), the event broker stops delivering messages to the client until the client acknowledges messages that have already been delivered.
: You can configure the maximum number of delivered‑but‑unacknowledged messages limit for a queue or topic endpoint provisioned on a Message VPN (refer to Message VPN-Level Guaranteed Messaging Configuration).
An event broker has a system-level limit for the maximum number delivered‑but‑unacknowledged messages for all clients at a given time. On reaching this maximum, no more messages are delivered to clients until some clients return acknowledgments back to the event broker, or they are disconnected. The maximum number delivered‑but‑unacknowledged messages supported is dependent on the event broker model.
By default, an event is generated when the number of outstanding messages that have not been acknowledged by their receiving clients reaches 80 percent of the system maximum (the set value), and this is followed by a further event the number of client-unacknowledged messages returns below 60 percent of the system maximum (the clear value). The thresholds for when these events are generated are configurable (refer to Delivered Unacked Thresholds).
To set a limit on the amount of time that published messages have to be consumed and acknowledged, you can assign Time‑to-Live (TTL) expiry values to them.
- If your application is using the Solace messaging APIs or REST service to publish Guaranteed messages, you can set a Time‑to-Live (TTL) expiry value on each published Guaranteed message to indicate how long the message should be considered valid. A publisher TTL expiration starts when a message is published and counts down as the message passes through the network.
- You can configure a Max Time to Live (TTL) value for a durable endpoint so that received messages are provided with expiration value to limit how long they can remain on that durable endpoint when a Max TTL is used. The Max TTL only applies when a message is on the endpoint.
When TTL values are applied to messages in either or both of these ways, messages that are not consumed and acknowledged before their expiration times are reached are discarded or moved to a Dead Message Queue (DMQ) . If a message has both a publisher-assigned TTL and an endpoint-assigned Max TTL, the event broker will use the minimum of the two TTL values when the message is on the endpoint.
: If a Message VPN bridge is used so that published messages that match topic subscriptions can be delivered from a remote Message VPN on one event broker to a local Message VPN on another event broker, the amount of time the message spends on each event broker is counted. That is, the amount of time a message spends on the remote event broker is counted, and its remaining time to live is updated when it is sent to the local Message VPN. For example, with a publisher-provided TTL of eight seconds, if a message spends two seconds on the first event broker, before it reaches the local Message VPN on the second event broker, it will have a TTL of six seconds.
Using TTLs to expire messages that have not been processed quickly can help prevent stale messages from being delivered to consumers. However, it should be noted that monitoring and processing messages using TTLs can affect the system-level limits for Guaranteed message delivery (for more information, refer to Guaranteed Message Queuing Limits).