Message Replay

Message replay allows event brokers to resend messages to new or existing clients that request them hours (or even days) after those messages were first received by the event broker. When message replay is enabled, event brokers store persistent messages in a replay log. Messages are retained in the replay log until its spool usage reaches 90% of its configured quota, after which the oldest messages are removed from the replay log to make space for new messages.

When message replay is enabled for a Message VPN, replay can be initiated for a non-partitioned queue or topic endpoint from the event broker management interfaces (PubSub+ Broker Manager, Solace CLI, and SEMP), or from client applications using the PubSub+ Messaging APIs. When you initiate message replay, the event broker delivers the messages from the replay log that match any subscription on the queue or topic endpoint, beginning from a specified start point, to the end of the replay log. The messages from the replay stream then join the stream of live messages without requiring any action from the consuming application.

For more information, see:

For instructions to configure or initiate message replay, see:

You can also watch the Message Replay Configuration Video showing how to get message replay set up quickly using PubSub+ Broker Manager.

Understanding Message Replay

Storing Messages for Replay

To enable message replay for a Message VPN you configure a replay log.

  • You can configure how many megabytes (MB) of spool usage the replay log consumes on each Message VPN by configuring the replay log's max-spool-usage.

  • By default, all Guaranteed messages that are not rejected to the publisher are stored in the replay log. You can use topic filters, including wildcard subscriptions and topic exceptions, to specify which messages are added to the replay log. The replay log maintains the original publication order across topics. For more information, see Filtering the Messages to Add to the Replay Log .

  • The event broker automatically trims the oldest messages from the replay log to make room for new messages as they arrive. Trimming occurs when the message replay spool reaches 90% of its configured quota. Optionally, you can manually trim the replay log to reduce the frequency of automatic trimming. For more information, see Trimming Message Replay Logs.

    Solace recommends that you monitor the replay log to ensure that the replay log does not exceed 90% of its configured quota for max-spool-usage. For more information, see Monitoring Message Replay.

  • If the replay log reaches 100% of its configured quota for max-spool-usage, the event broker stops receiving messages, and all unprocessed Guaranteed messages are retransmitted. In addition, rejected messages are not added to the replay log to prevent its spool usage from exceeding the configured quota.

  • Messages that are rejected (NACKed) to the publisher are not added to the replay log.

  • Message replay is not supported for partitioned queues.

  • Message replay is not supported with replication. Messages written to the replay log on the active site may not be written to the replay log at the standby site.

Replaying Messages

A message replay request can be initiated for an endpoint by either a client application using a PubSub+ Messaging API when it binds to a Guaranteed endpoint, or from Broker Manager, Solace CLI or SEMP. Message replay can be performed for any consumer, no matter what protocol or API they are using, including AMQP, MQTT, REST/WebHooks, and PubSub+ Messaging APIs.

Messages are replayed only to the endpoint that replay is requested for. If you initiate message replay from Broker Manager or Solace CLI, you have to specify the endpoint that you want to replay messages to.

Message replay supports the following types of playback requests:

Start replay from the beginning of the log
The requester specifies that the playback is to start from the oldest message in the replay log. All messages in the replay log that match the endpoint’s subscriptions are delivered to the endpoint.
Start replay from a specific date
The requester specifies the date and time from which the playback is to start. Any messages in the replay log equal to, or newer than, the specified date and time that match the endpoint’s subscriptions are delivered to the endpoint.
Start replay from replication group message ID
The requester specifies a replication group message ID after which the playback is to start. Any messages in the replay log received after the specified replication group message ID that match the endpoint's subscriptions are delivered to the endpoint.

When a replay begins, the replay proceeds as follows:

  • All logged messages from the playback starting point are processed and the event broker replays logged messages that match the replaying endpoint's subscriptions.

  • The event broker sends the original messages as they were initially received, without modification, including the original topic, content, and message ID. The event broker doesn't mark the message as redelivered in any way.

  • Messages are replayed in the original order they would have been sent to the client, even across different topics.

  • In all cases, once the replay has caught up to the live data stream, message delivery switches to live message delivery from the endpoint.

  • If a replay is initiated to an endpoint that has no clients connected to consume and acknowledge messages, replay sends a maximum 1000 messages to the endpoint to ensure that the unacknowledged resources of the endpoint are not completely consumed. For more information, see Resource Consumption . After a client connects and begins to consume messages from the endpoint, any remaining messages in the log are replayed to the endpoint.

  • If the endpoint has connected clients, the number of messages replayed is limited only by the Maximum Delivered Unacknowledged Messages per Flow (max-delivered-unacked-msgs-per-flow) setting on the endpoint (up to 100,000 messages).

  • Initiating a replay on an endpoint already under replay cancels the previous replay and starts a new one.

  • Live messages received while an endpoint is under replay are processed as usual, including being written to the replay log, except that they are not sent to the endpoint under replay. The endpoint receives the messages from the replay log in sequence. When replay completes, the endpoint begins receiving live messages.

  • A replication group message ID is an attribute of Solace messages, assigned by the event broker delivering the messages to the queue and topic endpoints, that uniquely identifies a message on a particular queue or topic endpoint within a high availability (HA) group and replication group of event brokers. In an event mesh, when event brokers are interconnected using either Dynamic Message Routing (DMR) or VPN bridges, a given message is assigned a different replication group message ID in each replication group in the mesh. When replaying after a specific replication group message ID, the replay request fails if the message ID specified can't be found in the replay log.

  • You can replay to a temporary topic endpoint or temporary queue provided that you configure the event broker appropriately. For more information, see Enabling Replay on Temporary Queues.

Additional Considerations When Using Message Replay

You should be aware of the following considerations for message replay:

  • Message replay consumes event broker processing resources because it needs to maintain the replay log, handle retrieval and replay of messages, and trim the replay log whenever its spool usage reaches 90% of its configured quota for max-spool-usage.

  • Message replay is intended to be used for Guaranteed messages. Direct messages can be logged only by promoting them to Guaranteed messages.

    Promoting Direct messages can dramatically affect the event broker's performance. A Direct message publisher can send messages faster than the Guaranteed messaging path can process messages. In this case, the event broker aggressively discards messages on the promotion path to protect the Guaranteed messaging path. If you have Direct messages that you want to record in the replay log, Solace recommends changing the publisher to use Guaranteed messages. For more information about message promotion and the associated risks, see Topic Matching and Message Delivery Modes.

  • Message replay is not supported for partitioned queues.

  • Slow disks negatively impact message replay performance.

  • Replication with message replay isn't supported. Messages written to the replay log on the active site may not be written to the replay log at the standby site.

  • Message replay is not supported for partitioned queues.

  • Replaying to any exclusive queue whose active consumer is using an egress selector to filter messages is not recommended. If an egress selector is used, and it causes replayed messages to stay indefinitely on a replaying endpoint, the replay won't complete.

Message Replay Walk-Through

The following diagram shows a simplified example of how message replay works on an event broker.

Diagram showing the flow described in the surrounding text

This walkthrough follows the journey of messages from a publisher, through an event broker, to a subscriber to explain how messages are logged for replay and played back at a later time.

  1. A publisher application connects to a Message VPN that had a replay log enabled and publishes messages.

  2. The event broker performs the following actions:

    1. Gives unique replication group message IDs to each message
    2. Writes each message to persistent storage
    3. Adds a reference to each message to the appropriate subscribing queues
    4. Adds a reference to each message to the replay log

  3. The subscriber performs the following actions:

    1. Consumes each message from their queue.
    2. Processes each message.
    3. Sends an acknowledgment for each message to the event broker.

  4. After receiving an acknowledgment, the event broker deletes the acknowledged message from the subscriber's queue.

  5. The message remains in the replay log until it is the oldest message in the log and needs to be trimmed to make space for newer messages.

  6. Some time later the subscriber experiences a system error and loses recent data. To retrieve messages that were delivered but lost, the subscriber requests a replay in one of two ways:

    • The subscribing application directly requests message replay using a PubSub+ Messaging API.
    • An administrator initiates message replay using PubSub+ Broker Manager,Solace CLI, or SEMP.

    The subscriber requests one of the following replay options:

    • Replay messages from a specific time if time of the data loss is known.
    • Replay messages following a specific replication group message ID if the last message ID that was retained is known.
    • Replay all messages available in the replay log, starting from the oldest message.

  7. The replayable messages are delivered to the replaying endpoint and, if the consumer is connected, to the consumer.

  8. The consumer acknowledges the replayed message, which removes it from the replaying endpoint; however, it remains in the replay log until it is trimmed.

Use Cases

Message replay can restore or recreate messages in the application databases in several situations.

Application Corruption Recovery

Consider an application that's connected to one or more queues, receives messages, processes them, updates its own persistent database to reflect message content, including the replication group message ID retrieved from the message, and acknowledges receipt of the message. This acknowledgment deletes the messages from the queues. If the application experiences some form of crash that results in the corruption of its own database, the application may be restarted with a known good database, perhaps from earlier in the day, but any work it has done in the meantime has been lost.

In this situation, the application might request message replay to its queues, starting from the replication group message ID stored in the last record in the database, redoing the work it had already done. During replay, new live messages will be added to the replay log. Once all logged messages have been replayed, the application automatically rejoins the live data stream.

The administrator can also initiate replay of a queue on the client’s behalf through Solace CLI, Broker Manager, or SEMP commands.

Application Misconfiguration Recovery

Consider an application where a set of administrator-configured topic-to-queue mappings is used to attract messages to the application’s queues. If a mistake is made in the configuration and the set of topics on the queue is not correct, the error may not be noticed right away.

In this case, the administrator corrects the topic-to-queue mappings on the queue, and restarts the application with a known good state or empty state. The administrator or application requests message replay to its queues, redoing its work with the now corrected set of topic-to-queue mappings. After all old messages, including messages that were missed due to the incorrect topic-to-queue-mapping, have been replayed to the application, it automatically rejoins the live data stream.

Application Late-Join

In this use-case, an application that did not have queues or subscriptions on the event broker at the time the data was published may wish to connect to the event broker, create its queues and subscriptions, and then use message replay to consume the previously published data. Typical applications for this use-case include bringing on-line a new analytics or trading algorithm that requires some amount of historical data to make decisions on the subsequent live data.

Event Sourcing and On-Demand Analytics

In this use-case, an enterprise may wish to replay the data that was sent to an application to capture the entire history of message flow to an application so that it can be reviewed or analyzed at a later date.