Message Replay

Message replay allows event brokers to resend messages to new or existing clients that request them hours (or even days) after those messages were first received by the event broker. When message replay is enabled, event brokers store the persistent messages in a replay log. These messages are retained in the replay log unit its spool usage reaches 90% of its configured quota, after which the oldest messages are removed from the replay log to make space for new messages through a process called trimming.

Message replay can be performed on a non-partitioned queue or topic endpoint. Replay can be initiated from the event broker management interfaces (PubSub+ Broker Manager, Solace CLI, and SEMP), or from client applications using the PubSub+ Messaging APIs. When initiating replay, you can request all messages in the replay log, all messages following a specified replication group message ID, or all messages starting from a requested replay start time. The event broker delivers the messages from the replay log that match any subscription on that queue or topic endpoint, from the specified start point to the end of the replay log. The messages from the replay stream then join the stream of live messages without requiring any action from the consuming application.

Message replay is not supported for partitioned queues.

Message replay is not supported with replication. Messages written to the replay log on the active site may not be written to the replay log at the standby site.

For more information about message replay, see:

• Understanding Message Replay
• Message Replay Walk-Through
• Use Cases
• Monitoring Message Replay
• Message Replay / PubSub+ Cache Comparison

If you're ready to use message replay, you can learn how to configure and use it by jumping to Message Replay Configuration. Alternatively, you can watch the Message Replay Configuration Video on how to get message replay set up quickly using PubSub+ Broker Manager.

Understanding Message Replay

You can configure which Message VPNs are configured with message replay.

Message Storage

The following summary describes how message replay uses message storage:

• You can configure how many megabtyes (MB) of spool usage the replay log consumes on each Message VPN by by configuring the replay log's max-spool-usage. We refer to this as the message replay's configured quota for max-spool-usage.

• By default, all Guaranteed messages that are not rejected to the publisher are stored in the replay log. You can use topic filters, including wildcard subscriptions and topic exceptions, to select which messages are added to the replay log. The replay log maintains the original publication order across many topics. For more information, see Specifying the Messages to Add to the Replay Log .

• The event broker automatically trims the oldest messages from the replay log to make room for new messages as they arrive. Trimming occurs when the message replay spool reaches 90% of its configured quota. Optionally, you can choose to manually trim the replay log to reduce the frequency of automatic trimming. For more information, see Trimming the Replay Log.

  Solace recommends that you monitor the replay log to ensure that the replay log does not exceed 90% of its configured quota for max-spool-usage. For more information, see Monitoring Message Replay.

• If the replay log reaches 100% of its configured quota for max-spool-usage , the event broker stops receiving messages, and all unprocessed guaranteed messages are retransmitted. In addition, rejected messages are not added to the replay log to prevent its spool usage from exceeding the configured quota.

Replaying Messages

• A replay can begin from a specific time, after a specific replication group message ID, or from the beginning of the message replay log.

• When a replay begins, all logged messages from the playback starting point are processed and the event broker replays logged message that match the replaying endpoint's subscriptions. The event broker sends the original message as it was initially received, without modification, including the original topic, content, and message ID. The event broker doesn't mark the message as redelivered in any way.

• Messages are replayed in the original order they would have been sent to the client, even across different topics.

• Message replay can be performed for any consumer, no matter what protocol or API they are using, including AMQP, MQTT, REST/WebHooks, and PubSub+ Messaging APIs.

• Messages are replayed only to the endpoint for which they are requested. If you initiate message replay from Broker Manager or Solace CLI you have to specify the endpoint that you want to replay messages to.

• If a replay is initiated to an endpoint that has no clients connected to consume and acknowledge messages, replay sends a maximum 1000 messages to the endpoint to ensure that the unacknowledged resources of the endpoint are not completely consumed. For more information, see Resource Consumption . After a client connects and begins to consume messages from the endpoint, any remaining messages in the log are replayed to the endpoint.

• If the endpoint has connected clients, the number of messages replayed is limited only by the Maximum Delivered Unacknowledged Messages per Flow (max-delivered-unacked-msgs-per-flow) setting on the endpoint (up to 100,000 messages).

• Initiating a replay on an endpoint already under replay cancels the previous replay and starts a new one.

• Live messages received while an endpoint is under replay are processed as usual, including being written to the replay log, except that they are not sent to the endpoint under replay. The endpoint receives the messages from the replay log in sequence. When replay completes, the endpoint begins receiving live messages.

• A replication group message ID is an attribute of Solace messages, assigned by the event broker delivering the messages to the queue and topic endpoints, that uniquely identifies a message on a particular queue or topic endpoint within a high availability (HA) group and replication group of event brokers.

A replication group message ID only uniquely identifies a message within a high-availability (HA) group or replication group. In an event mesh, when event brokers are interconnected using either Dynamic Message Routing (DMR) or VPN bridges, a given message is assigned a different replication group message ID in each replication group in the mesh. When replaying after a specific replication group message ID, the replay request fails if the message ID specified can't be found in the replay log.

Message Replay Walk-Through

The following diagram shows a simplified example of how message replay works on an event brokers and to help you understand the flow of messages.

Basic Message Flow

 

This walkthrough follows the journey of messages from a publisher, through an event broker, to finally a subscriber to show you how messages are logged for replay and played back at a later time.

1. A publisher application connects to the event broker and publishes messages.

2. The event broker performs the following actions:

   1. Gives unique replication group message IDs to each message

   2. Writes each message to persistent storage

   3. Adds a reference to each message to the appropriate subscribing queues

   4. Adds a reference to each message to the replay log

3. The subscriber performs the following actions:

   1. Consumes each message from their queue.

   2. Processes each message.

   3. Sends an acknowledgment for each message to the event broker.

4. After receiving an acknowledgment, the event broker deletes the acknowledged message from the subscriber's queue.

5. The message remains in the replay log until it is the oldest message in the log and needs to be trimmed to make space for newer messages.

6. Some time later the subscriber experiences a system error and loses recent data. To retrieve messages that were delivered but lost, the subscriber requests a replay in one of two ways:

   • The subscribing application directly requests message replay using a PubSub+ Messaging API.

   • An administrator initiates message replay using PubSub+ Broker Manager,Solace CLI, or SEMP.

   The subscriber requests one of the following replay options:

   • Replay messages from a specific time if time of the data loss is known.

   • Replay messages following a specific replication group message ID if the last message ID that was retained is known.

   • Replay all messages available in the replay log, starting from the oldest message.

Use Cases

Message replay can restore or recreate messages in the application databases in several situations.

Application Corruption Recovery

Consider an application that's connected to one or more queues, receives messages, processes them, updates its own persistent database to reflect message content, including the replication group message ID retrieved from the message, and acknowledges receipt of the message. This acknowledgment deletes the messages from the queues. If the application experiences some form of crash that results in the corruption of its own database, the application may be restarted with a known good database, perhaps from earlier in the day, but any work it has done in the meantime has been lost.

In this situation, the application might request message replay to its queues, starting from the replication group message ID stored in the last record in the database, redoing the work it had already done. During replay, new live messages will be added to the replay log. Once all logged messages have been replayed, the application automatically rejoins the live data stream.

The administrator can also initiate replay of a queue on the client’s behalf through Solace CLI, Broker Manager, or SEMP commands.

Application Misconfiguration Recovery

Consider an application where a set of administrator-configured topic-to-queue mappings is used to attract messages to the application’s queues. If a mistake is made in the configuration and the set of topics on the queue is not correct, the error may not be noticed right away.

In this case, the administrator corrects the topic-to-queue mappings on the queue, and restarts the application with a known good state or empty state. The administrator or application requests message replay to its queues, redoing its work with the now corrected set of topic-to-queue mappings. After all old messages, including messages that were missed due to the incorrect topic-to-queue-mapping, have been replayed to the application, it automatically rejoins the live data stream.

Application Late-Join

In this use-case, an application that did not have queues or subscriptions on the event broker at the time the data was published may wish to connect to the event broker, create its queues and subscriptions, and then use message replay to consume the previously published data. Typical applications for this use-case include bringing on-line a new analytics or trading algorithm that requires some amount of historical data to make decisions on the subsequent live data.

Event Sourcing and On-Demand Analytics

In this use-case, an enterprise may wish to replay the data that was sent to an application to capture the entire history of message flow to an application so that it can be reviewed or analyzed at a later date.

Message Replay / PubSub+ Cache Comparison

Message replay isn't a replacement for PubSub+ Cache. There are a number of differences between the two, and they're listed in the following table. Whether you choose message replay or PubSub+ Cache depends on your use cases.

Attribute	Message Replay	PubSub+ Cache
Use Case	Message Log Records all Guaranteed messages published to the Message VPN. Automatically trims oldest messages from log when the replays log exceeds 90% of its configured quota for max-spool-usage	Last Value Cache Typically keeps only the last message published on each topic. While PubSub+ Cache can be configured to keep more than one message per topic, it is a less common use case.
Message QoS	Logs Guaranteed (or persistent) messages. Logs any messages that get promoted to Guaranteed messages.	Intended for caching Direct (or non-persistent) messages. Connects to the event broker using Direct messaging. If any Guaranteed messages match PubSub+ Cache subscriptions, those messages are demoted to Direct when sent to PubSub+ Cache.
Interactions With Client API	All interactions with client APIs use Guaranteed messaging.	All interactions with client APIs use Direct messaging.
Topics Recorded	Logs all Guaranteed messages published in the Message VPN by default. You can also configure message replay to log messages that match a set of wildcard subscriptions and filters.	Administrator configures the wildcard topics that are to be stored in the cache.
Message Delivery Order	Replays messages in original published order. Publish order between topics is preserved.	Cache requests are fulfilled topic-by-topic. Order is preserved within a topic (if depth is greater than 1), but original publishing order between topics is not preserved.
Replay Request Mechanism	PubSub+ Messaging APIs Management interface	PubSub+ Messaging APIs
Deployment Model	Integrated feature of the PubSub+ event broker.	Software application that runs on an external Linux server.
Message Storage	Uses the Guaranteed messaging infrastructure to persistently store messages.	Stores cached messages in RAM.
Redundancy Model	Leverages the event broker’s active-standby high-availability (HA ) model.	Deploy multiple instances for N+1 redundancy.
Performance and Scaling	Built for logging messages at Guaranteed messaging rates. Add more PubSub+ event brokers to horizontally scale Guaranteed messaging and message replay.	Built for very high rate Direct messaging use-cases like market data distribution. Supports deploying multiple instances to horizontally scale.
Availability	Integrated into the PubSub+ event broker; no extra license.	Optional add-on product that requires purchased license.