Message Replay

Message Replay is a Solace PubSub+ feature that allows an event broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the event broker.

With replay enabled, event brokers store persistent messages in a replay log. These messages are kept until the log is full, after which the oldest messages are removed to free-up space for new messages.

Replay can be performed on a (non-partitioned) queue or topic endpoint. When initiated, you can request all messages in the replay log, all messages following a specified replication group message ID, or all messages starting from a requested replay start time. From the requested starting point, the event broker will deliver, to the queue or topic endpoint, the messages from the replay log that match any subscription on that queue or topic endpoint.

Replay can be initiated by event broker commands or client applications. Messages can be replayed starting from a specific date and time, after a specific replication group message ID, or as far back as the replay log extends. The replay stream eventually joins the stream of live messages without requiring any action from the consuming application.

Replay functionality helps protect applications by giving them the ability to correct database issues they might have arising from events such as misconfiguration, application crashes, or corruption, by redelivering data to the applications.

When replaying after a specific replication group message ID, the replay request will fail if the message ID specified cannot be found in the replay log.

Message replay is not supported for partitioned queues.

Message replay is not supported with replication. Messages written to the replay log on the active site may not be written to the replay log at the standby site.

Key Concepts

A replay can be specified to begin from a specific time, after a specific replication group message ID, or from the beginning of the message replay log.
A replication group message ID is an attribute of Solace messages, assigned by the event broker delivering the messages to the queue and topic endpoints, that uniquely identifies a message on a particular queue or topic endpoint within an HA group and replication group of event brokers.
You can initiate a replay from all Solace APIs or administratively for non-Solace protocols via Solace CLI, SEMP or PubSub+ Broker Manager.
- Message Replay can be performed for any consumer no matter what protocol or API they are using, including AMQP, MQTT, REST/WebHooks and Solace APIs.
Wildcard subscriptions on queues and topic endpoints can be used to select which messages from the replay log should be replayed while still maintaining the original publication order across these many topics.
Messages are replayed in the original order they would have been sent to the client – even across different topics.
You can enable and disable Message Replay on a per Message VPN basis.
You can configure the storage depth of the replay log on a per Message VPN basis.
The replay log is automatically pruned as it reaches capacity.

A replication group message ID can only identify a message within an HA group or replication group. In an event mesh, when brokers are interconnected using either DMR or VPN bridges, a given message will be assigned a different replication group message ID in each replication group in the mesh.

Next Steps

From here there are a few paths you can take depending on what you want to learn,

You can go to the Walkthrough section for a high-level, illustrated example of the basic operation of Message Replay.
If you're interested in learning more about how Message Replay can be used in your network, consider reviewing the Message Replay Use Cases.
If you're ready to use Message Replay, you can begin learning about how to configure and use it by jumping to Message Replay Configuration.
And there's a short video on how to get Message Replay up-and-running in under 4 minutes which you can view here:

Walkthrough

In this section we'll take a high-level look at how Message Replay works on a Solace PubSub+ event broker. We won't go into every detail just yet, just walkthrough the basic ideas via a simplified example so you can get a sense of how Message Replay works. If you're ready for a deep dive into details and configuration, take a look at Message Replay Configuration.

Basic Operation

Diagram

The way that Message Replay works is fairly straightforward. Let's follow the journey of messages from a Publisher, through the Solace PubSub+ event broker, to a Subscriber, and see how they get logged for future playback.

Message publication

Consider that you have a Publisher application that connects to the event broker and publishes messages. Those messages are given unique replication group message IDs by the broker, get added to the Queues that are subscribing, and they also get written to the Replay Log. One thing that's important to keep in mind is that the event broker isn't actually writing two copies of each message to the persistent store, only one copy is written, and the event broker keeps two references to it: there's one in the queue that's going to deliver the message to the Subscriber, and a second that's in the Replay Log.

Message consumption

The Subscriber consumes the messages, processes them, and acknowledges them. If processing includes writing the data to a database, the application may extract the replication group message ID from the message, and write that ID to the database along with the message data. The messages are then deleted from the Subscriber's queue, but they remain in the Replay Log. The Replay Log continues to store messages even after they have been delivered to all Subscribers.

Replaying messages

If some time later the Subscriber wants to get back those messages that have already been delivered, it can request a replay. The messages will be fetched from the Replay Log, and they'll be delivered to the Subscriber a second time.

You can choose to replay messages from a specific time if you know what time you want to start the replay at, or following a specific replication group message ID if you know the last message ID that was successfully processed, or you can start replay from the oldest message available in the Replay Log.

There are two ways of initiating a message replay:

If your application is using a Solace API, then you can kick off the replay request from your application.
Your other alternative is to initiate replay from Broker Manager, Solace CLI, or SEMP. Using this administration kick-off method allows your application to use replay regardless of the protocol or API you are using.

Some Other Key Characteristics

Wildcards

One of the most powerful features of Message Replay is the ability to replay messages not just to a fully qualified topic, but to a wildcard topic, or to a wildcard subscription. For example, you could have a queue set up to receive messages on topics orders/>, request a replay, and the event broker will deliver all the messages in the Replay Log that match the orders/> wildcard subscription. This makes Message Replay quite flexible in terms of what can be delivered from the Replay Log to an application.

Messages are only replayed to the endpoint for which the replay is requested

If a replay is initiated from Broker Manager, or Solace CLI, then one of the things you have to specify is which endpoint you want to replay messages to.

For example, if you're initiating replay from a Solace API on a queue, then the queue that your application is connected to is where the replay is going to happen. This is important because you may have other queues on your event broker that subscribe to that same topic, and those normally functioning applications would be very confused if they suddenly started receiving replay data.

Limits on number of messages sent in a replay request

If a replay is initiated to an endpoint that has no clients connected, only 1000 messages are replayed. This ensures that the unacked resources of the endpoint are not completely consumed.

If there are connected clients, the number of messages replayed is limited only by the max-delivered-unacked-msgs-per-flow setting on the endpoint (up to 100,000 messages).

For more information, see Resource Consumption .

Resource configuration

Not surprisingly Message Replay does consume resources on a PubSub+ event broker, so as part of configuring the event broker you'll choose which of the Message VPNs will have replay enabled, and which Message VPNs don't need this capability, so you can disable Message Replay on those other Message VPNs.

As part of setting up Message Replay, you can also specify how many gigabytes of persistent store you want your Replay Log to consume inside of a Message VPN.

Replay Log pruning

Once the Replay Log grows to the maximum depth you've configured for it, then the event broker will automatically prune out the oldest messages from the Replay Log so that there's room for new messages as they arrive to be written to the Replay Log

Use Cases

In this section we’ll look at a number of use cases where Message Replay can restore or recreate application databases.

Application Corruption Recovery

Consider an application that's connected to one or more queues, receives messages, processes them, updates its own persistent database to reflect message content including the replication group message ID retrieved from the message, and acknowledges receipt. This acknowledgment deletes the messages from the queue(s). Say the application then experiences some form of crash which results in the corruption of its own database. The application may be restarted with a known good database, perhaps from earlier in the day, but any work it has done in the meantime has been lost.

In this situation, the application might request message replay from its queues from the replication group message ID stored in the last record in the database, redoing the work it had already done. During replay, new live messages will be added to the replay log. Once all logged messages have been replayed, the application automatically rejoins the live data stream.

The administrator may also initiate replay of a queue on the client’s behalf through Solace CLI, Broker Manager, or SEMP commands.

Application Misconfiguration Recovery

Continuing with the application in the previous example, where a set of administrator configured topic-to-queue mappings is used to attract messages to the application’s queues. In this case a mistake is made and the set of topics on the queue is not correct, and the error is not noticed until part way through the business day.

In this case, the administrator corrects the topic-to-queue mappings on the queue, and restarts the application with a known good state from start-of-day, or last checkpoint, or beginning-of-replay-log. The administrator or application requests message replay from its queues, redoing its work with the now corrected set of topic-to-queue mappings in play. Once all old messages, including ones that were missed due to the incorrect topic-to-queue-mapping, have been replayed to the application it automatically rejoins the live data stream.

Application Late-Join

In this use-case, an application which did not have queues or subscriptions on the event broker at the time the data was published may wish to connect to the event broker, create its queues and subscriptions, and then replay and consume the previously published data. Typical applications for this use-case include bringing on-line a new analytics or trading algorithm which requires some amount of historical data to make decisions on the subsequent live data.

Event Sourcing & On-Demand Analytics

In this use-case, an enterprise may wish to capture the entire history of message flow to an application so that it can be reviewed or analyzed at a later date by replaying the data that was sent to the application.

Message Replay / PubSub+ Cache Comparison

Message Replay isn't a replacement for PubSub+ Cache. There are a number of differences between the two, and they're listed in the following table. Whether you choose Message Replay or PubSub+ Cache depends on your use cases.

Attribute	Message Replay	PubSub+ Cache
Use Case	Message Log Records all Guaranteed messages published to the Message-VPN. Automatically trims oldest messages from log when log’s quota is reached.	Last Value Cache Typically keeps only the last message published on each topic. While PubSub+ Cache can be configured to keep more than one message per topic, that is a less common use-case.
Message QoS	Logs Guaranteed (aka Persistent) messages. Logs any messages that get promoted to Guaranteed messages.	Intended for caching Direct (aka Non-Persistent) messages. Connects to the event broker using Direct messaging. If any Guaranteed messages match PubSub+ Cache subscriptions, those messages are demoted to Direct when sent to PubSub+ Cache.
Interactions With Client API	All interactions with client APIs use Guaranteed messaging.	All interactions with client APIs use Direct messaging.
Topics Recorded	Logs all Guaranteed messages published in the Message-VPN.	Administrator configures the wildcard topics that are to be stored in the cache.
Message Delivery Order	Replays messages in original published order. Publish order between topics is preserved.	Cache requests are fulfilled topic-by-topic. Order is preserved within a topic (if depth is greater than 1), but original publish order between topics is not preserved.
Replay Request Mechanism	Solace APIs Management Interface	Solace APIs
Deployment Model	Integrated feature of the PubSub+ event broker.	Software application that runs on an external Linux server.
Message Storage	Uses the Guaranteed messaging infrastructure to persistently store messages.	Stores cached messages in RAM.
Redundancy Model	Leverages the event broker’s active-standby HA model.	Deploy multiple instances for n+1 redundancy.
Performance and Scaling	Built for logging messages at Guaranteed messaging rates. Add more PubSub+ event brokers to horizontally scale Guaranteed messaging and Message Replay.	Built for very high rate Direct messaging use-cases like market data distribution. Supports deploying multiple instances to horizontally scale.
Availability	Integrated into the PubSub+ event broker; no extra license.	Optional add-on product that requires purchased license.

Provide feedback