Message Replay

Message replay allows event brokers to resend messages to new or existing clients that request them, hours, or even days, after those messages were first received by the event broker.

With message replay enabled, event brokers store persistent messages in a replay log. These messages are kept until the log is full, after which the oldest messages are removed as needed to make space for new messages.

Replay can be performed on a non-partitioned queue or topic endpoint. When initiating replay, you can request all messages in the replay log, all messages following a specified replication group message ID, or all messages starting from a requested replay start time. The event broker delivers the messages from the replay log that match any subscription on that queue or topic endpoint, from the specified starting point to the end of the replay log.

Replay can be initiated from the event broker management interfaces (PubSub+ Broker Manager, Solace CLI, and SEMP), or from client applications using the Solace messaging APIs. Messages can be replayed starting from a specific date and time, after a specific replication group message ID, or as far back as the replay log extends. The replay stream eventually joins the stream of live messages without requiring any action from the consuming application.

Message replay is not supported for partitioned queues.

Message replay is not supported with replication. Messages written to the replay log on the active site may not be written to the replay log at the standby site.

For more information about message replay, see the following sections:

If you're ready to use message replay, you can learn how to configure and use it by jumping to Message Replay Configuration, or watch the Message Replay Configuration Video on how to get message replay set up quickly using PubSub+ Broker Manager.

Understanding Message Replay

The following key concepts explain message replay:

General Concepts

  • You can enable and disable the replay log for each Message VPN. The replay log consumes resources on the event broker. You can choose which Message VPNs have a replay log, and which Message VPNs don't need this capability.

  • You can initiate message replay from all Solace messaging APIs or administratively for other protocols via Solace CLI, SEMP or PubSub+ Broker Manager.

Message Storage

  • You can configure the how many gigabytes of storage the replay log consumes on each Message VPN.

  • Wildcard subscriptions can be used to select which messages are added to the replay log while still maintaining the original publication order across these many topics.

  • After the replay log grows to the maximum configured size, the event broker automatically trims the oldest messages from the replay log to make room for new messages as they arrive.

  • Messages that send a negative acknowledgment to the publisher are not added to the replay log.

Replaying Messages

  • A replay can begin from a specific time, after a specific replication group message ID, or from the beginning of the message replay log.

  • When a replay begins, all logged messages from the playback starting point are processed, and the event broker replays logged message that match the replaying endpoint's subscriptions. The event broker sends the original message as it was initially received, without modification, including the original topic, content, and message ID. The event broker doesn't mark the message as redelivered in any way.

  • Messages are replayed in the original order they would have been sent to the client – even across different topics.

  • Message replay can be performed for any consumer no matter what protocol or API they are using, including AMQP, MQTT, REST/WebHooks, and Solace messaging APIs.

  • Messages are replayed only to the endpoint for which they are requested. If you initiate message replay from Broker Manager or Solace CLI you have to specify the endpoint that you want to replay messages to.

  • If a replay is initiated to an endpoint that has no clients connected to consume and acknowledge messages, replay sends a maximum 1000 messages to the endpoint to ensure that the unacknowledged resources of the endpoint are not completely consumed. For more information, see Resource Consumption . After a client connects and begins to consume messages from the endpoint, any remaining messages in the log are replayed to the endpoint.

  • If the endpoint has connected clients, the number of messages replayed is limited only by the max-delivered-unacked-msgs-per-flow setting on the endpoint (up to 100,000 messages).

  • Initiating a replay on an endpoint already under replay cancels the previous replay and starts a new one.

  • Live messages received while an endpoint is under replay are processed as usual, including being written to the replay log, except that they are not sent to the endpoint under replay. The endpoint receives the messages from the replay log in sequence. When replay completes, the endpoint begins receiving live messages.

  • A replication group message ID is an attribute of Solace messages, assigned by the event broker delivering the messages to the queue and topic endpoints, that uniquely identifies a message on a particular queue or topic endpoint within a high availability (HA) group and replication group of event brokers.

A replication group message ID only identifies a message within an HA group or replication group. In an event mesh, when brokers are interconnected using either dynamic message routing (DMR) or VPN bridges, a given message is assigned a different replication group message ID in each replication group in the mesh. When replaying after a specific replication group message ID, the replay request will fail if the message ID specified can't be found in the replay log.

Message Replay Walkthrough

This section takes a high-level look at how message replay works on a Solace PubSub+ event broker using a simplified example to help you understand how message replay works.

Basic Message Flow

 

Diagram showing the flow described in the surrounding text

This walkthrough follows the journey of messages from a publisher, through event broker, to a subscriber to show how messages can be logged for replay and played back at a later time.

  1. A publisher application connects to the event broker and publishes messages.

  2. The event broker performs the following actions:

    1. gives unique replication group message IDs to each message

    2. writes each message to persistent storage

    3. adds a reference to each message to the appropriate subscribing queues

    4. adds a reference to each message to the replay log

  3. The subscriber performs the following actions:

    1. consumes each message from their queue,

    2. processes each message. If processing includes writing the data to a database, the application may extract the replication group message ID from the message, and write that ID to the database along with the message data.

    3. sends an acknowledgment for each message to the event broker

  4. After receiving an acknowledgment, the event broker deletes the acknowledged message from the subscriber's queue.

  5. The message remains in the replay log until it is the oldest message in the log and needs to be trimmed to make space for newer messages.

  6. Some time later the subscriber experiences a system error and loses recent data. To retrieve messages that were delivered but lost, the subscriber requests a replay in one of two ways:

    • The subscribing application directly requests message replay using a Solace API.

    • An administrator initiates message replay using Broker Manager,Solace CLI, or SEMP.

    The subscriber requests one of the following replay options:

    • Replay messages from a specific time if time of the data loss is known.

    • Replay messages following a specific replication group message ID if the last message ID that was retained is known

    • Replay all messages available in the replay log, starting from the oldest message

Use Cases

Message replay can restore or recreate messages in the application databases in several situations.

Application Corruption Recovery

Consider an application that's connected to one or more queues, receives messages, processes them, updates its own persistent database to reflect message content, including the replication group message ID retrieved from the message, and acknowledges receipt. This acknowledgment deletes the messages from the queues. If the application experiences some form of crash that results in the corruption of its own database, the application may be restarted with a known good database, perhaps from earlier in the day, but any work it has done in the meantime has been lost.

In this situation, the application might request message replay to its queues, starting from the replication group message ID stored in the last record in the database, redoing the work it had already done. During replay, new live messages will be added to the replay log. Once all logged messages have been replayed, the application automatically rejoins the live data stream.

The administrator can also initiate replay of a queue on the client’s behalf through Solace CLI, Broker Manager, or SEMP commands.

Application Misconfiguration Recovery

Consider an application where a set of administrator-configured topic-to-queue mappings is used to attract messages to the application’s queues. If a mistake is made in the configuration and the set of topics on the queue is not correct, the error may not be noticed right away.

In this case, the administrator corrects the topic-to-queue mappings on the queue, and restarts the application with a known good state or empty state. The administrator or application requests message replay to its queues, redoing its work with the now corrected set of topic-to-queue mappings. After all old messages, including messages that were missed due to the incorrect topic-to-queue-mapping, have been replayed to the application, it automatically rejoins the live data stream.

Application Late-Join

In this use-case, an application that did not have queues or subscriptions on the event broker at the time the data was published may wish to connect to the event broker, create its queues and subscriptions, and then use message replay to consume the previously published data. Typical applications for this use-case include bringing on-line a new analytics or trading algorithm that requires some amount of historical data to make decisions on the subsequent live data.

Event Sourcing and On-Demand Analytics

In this use-case, an enterprise may wish to replay the data that was sent to an application to capture the entire history of message flow to an application so that it can be reviewed or analyzed at a later date.

Message Replay / PubSub+ Cache Comparison

Message replay isn't a replacement for PubSub+ Cache. There are a number of differences between the two, and they're listed in the following table. Whether you choose message replay or PubSub+ Cache depends on your use cases.

Attribute Message Replay PubSub+ Cache
Use Case

Message Log

Records all Guaranteed messages published to the Message-VPN.

Automatically trims oldest messages from log when log’s quota is reached.

Last Value Cache

Typically keeps only the last message published on each topic.

While PubSub+ Cache can be configured to keep more than one message per topic, that is a less common use-case.

Message QoS

Logs Guaranteed (aka Persistent) messages.

Logs any messages that get promoted to Guaranteed messages.

Intended for caching Direct (aka Non-Persistent) messages.

Connects to the event broker using Direct messaging. If any Guaranteed messages match PubSub+ Cache subscriptions, those messages are demoted to Direct when sent to PubSub+ Cache.

Interactions With Client API All interactions with client APIs use Guaranteed messaging. All interactions with client APIs use Direct messaging.
Topics Recorded Logs all Guaranteed messages published in the Message-VPN. Administrator configures the wildcard topics that are to be stored in the cache.
Message Delivery Order

Replays messages in original published order.

Publish order between topics is preserved.

Cache requests are fulfilled topic-by-topic.

Order is preserved within a topic (if depth is greater than 1), but original publish order between topics is not preserved.

Replay Request Mechanism

Solace APIs

Management Interface

Solace APIs
Deployment Model Integrated feature of the PubSub+ event broker. Software application that runs on an external Linux server.
Message Storage Uses the Guaranteed messaging infrastructure to persistently store messages. Stores cached messages in RAM.
Redundancy Model Leverages the event broker’s active-standby HA model. Deploy multiple instances for n+1 redundancy.
Performance and Scaling

Built for logging messages at Guaranteed messaging rates.

Add more PubSub+ event brokers to horizontally scale Guaranteed messaging and message replay.

Built for very high rate Direct messaging use-cases like market data distribution.

Supports deploying multiple instances to horizontally scale.

Availability Integrated into the PubSub+ event broker; no extra license. Optional add-on product that requires purchased license.