When an event broker receives a Guaranteed message, it processes the message to determine if there are any registered topic subscriptions or queues that match the topic the message was published to. If there is a topic subscription match or a matching queue on the event broker, the message and all the matches are spooled, and then the event broker acknowledges receipt of the message. After acknowledging the message, the event broker attempts to deliver it to all the matching clients and event brokers. As each client and event broker acknowledges receipt of the message, the associated match is deleted from the match list of the message. Once there are no matches left associated with a message, the message itself is deleted from the spool.
If one or more of the clients are offline or have fallen behind, the message is held in persistent storage until it can be delivered. If there are too many messages to hold in memory, messages are written to the disk in large blocks. This means only messages for slow or offline clients need to be written to the disk, and they can be written in an efficient manner. If all of the clients acknowledge the message quickly enough, the message doesn't need to be written to disk.
A message may not be spooled because of resource or operating limitations. The event broker performs a variety of checks before a message is spooled. These include:
- Would spooling the message exceed the event broker-wide message spool quota?
- Would spooling the message exceed the Message VPN’s message spool quota?
- Would spooling the message exceed the endpoint’s message spool quota?
- Would spooling the message exceed the endpoint’s maximum permitted message size?
- Would the message exceed the endpoint’s maximum message size?
- Is the destination endpoint shutdown?
- If the message is a low-priority message, would spooling the message exceed the endpoint’s reject low‑priority message limit?
Depending on the reason for why the message was not spooled to the endpoint, either no acknowledgment is returned to the publisher, or a negative acknowledgment (that is, a ‘nack’) is returned, and it's up to the publisher to handle these possibilities. A statistic is then incremented on the event broker.
- If a subscription is deleted when a message is spooled for that subscription, the message will still be delivered.
- If an endpoint is deleted (for example, through the Solace CLI) while a message is spooled to that endpoint, the message will not be delivered.
Message Spool Files
If an
For more information, see Configuring Message Spool Sizes for PubSub+ Cloud and Spool Files Thresholds for software event brokers and appliances.
Windowed Acknowledgment
A windowed acknowledgment mechanism is used at the transport level between the event broker and individual clients publishing and receiving Guaranteed messages.
A windowed acknowledgment prevents the round-trip acknowledgment time from becoming the limiting factor for message throughput. It does this by allowing a configurable number of messages to be in transit between an event broker and a publishing or subscribing client before an acknowledgment is required. The window size can be configured through a Solace messaging API flow property.
Solace APIs also batch acknowledgments from the application to the event broker. The figure below shows a client application sending multiple acknowledgments for Guaranteed messages, which the API consolidates into a single acknowledgment on the wire that is returned to the event broker. The size of the batch is configured through an API flow property.
With any windowed acknowledgment scheme there is the possibility of failure in the time between a message being received by the client, and the time at which the event broker receives the acknowledgment. A failure at this time requires all non-acknowledged messages in transit to be sent again. Thus, the number of messages redelivered increases and is directly proportional to the combination of window size and acknowledgment threshold.
Cumulative ACK and Acknowledgment Thresholds
Delivered-But-Unacknowledged Messages
There is a hard limit for the number of Guaranteed messages that can be delivered through a consumer flow to a bound client without that client returning an acknowledgment for those messages. On reaching this limit
You can configure the maximum number of delivered‑but‑unacknowledged messages limit for a queue or topic endpoint provisioned on a Message VPN (refer to Message VPN-Level Guaranteed Messaging Configuration).
An event broker has a system-level limit for the maximum number delivered‑but‑unacknowledged messages for all clients at a given time. On reaching this maximum, no more messages are delivered to clients until some clients return acknowledgments back to the event broker, or they are disconnected. The maximum number delivered‑but‑unacknowledged messages supported is dependent on the event broker version.
By default, an event is generated when the number of outstanding messages that have not been acknowledged by their receiving clients reaches 80 percent of the system maximum (the set value), and this is followed by a further event the number of client-unacknowledged messages returns below 60 percent of the system maximum (the clear value).
Message Expiry
To set a limit on the amount of time that published messages have to be consumed and acknowledged, you can assign Time‑to-Live (TTL) expiry values to them.
- If your application is using the Solace messaging APIs or REST service to publish Guaranteed messages, you can set a Time‑to-Live (TTL) expiry value on each published Guaranteed message to indicate how long the message should be considered valid. A publisher TTL expiration starts when a message is published and counts down as the message passes through the network.
- You can configure a Max Time to Live (TTL) value for a durable endpoint so that received messages are provided with expiration value to limit how long they can remain on that durable endpoint when a Max TTL is used. The Max TTL only applies when a message is on the endpoint.
When TTL values are applied to messages in either or both of these ways, messages that are not consumed and acknowledged before their expiration times are reached are discarded or moved to a Dead Message Queue (DMQ) . If a message has both a publisher-assigned TTL and an endpoint-assigned Max TTL, the event broker will use the minimum of the two TTL values when the message is on the endpoint.
If a Message VPN bridge is used so that published messages that match topic subscriptions can be delivered from a remote Message VPN on one event broker to a local Message VPN on another event broker, the amount of time the message spends on each event broker is counted. That is, the amount of time a message spends on the remote event broker is counted, and its remaining time to live is updated when it is sent to the local Message VPN. For example, with a publisher-provided TTL of eight seconds, if a message spends two seconds on the first event broker, before it reaches the local Message VPN on the second event broker, it will have a TTL of six seconds.
Using TTLs to expire messages that have not been processed quickly can help prevent stale messages from being delivered to consumers. However, it should be noted that monitoring and processing messages using TTLs can affect the system-level limits for Guaranteed message delivery
Maximum Spool Usage
You can set a maximum value for the amount of spool disk space (max-spool-usage) that can be consumed by various objects on the event broker. The following tables list the default settings for this value and for the corresponding event thresholds.
System-Level Settings
| Broker Type | Max Spool Usage (MB) | Set Threshold (%) | Clear Threshold (%) |
|---|---|---|---|
|
Software Event Broker |
Default: 1,500 Maximum: 6,000,000 |
80 |
60 |
|
Appliance (3530) |
Default: 60,000 Maximum: 800,000 |
80 |
60 |
|
Appliance (3560) |
Default: 60,000 Maximum: 6,000,000 |
80 |
60 |
Message VPN Settings
| Broker Type | Max Spool Usage (MB) | Set Threshold (%) | Clear Threshold (%) |
|---|---|---|---|
|
All |
Default: 0 Maximum: Same as the maximum for the System-level setting for |
80 |
60 |
Settings for Queues / Queue Templates / Topic Endpoints / Topic Endpoint Templates / Client-Provisioned Endpoints
| Broker Type | Max Spool Usage (MB) | Set Threshold (%) | Clear Threshold (%) | Set Threshold (MB) | Clear Threshold (MB) |
|---|---|---|---|---|---|
|
All |
Default: 5000 Maximum: Same as the maximum for the System-level setting for |
25 |
18 |
1250 |
900 |
See the following links for information on setting max-spool-usage values:
See the following links for information on setting the event thresholds for message spool usage: