Trimming Message Replay Logs

As the replay log grows larger and reaches the configured capacity, older logged messages are deleted to make room for new ones. This process is called replay log trimming. The event broker automatically performs trimming when the replay log exceeds 90% of its configured maximum spool usage quota.

An endpoint can contain and successfully deliver replayed messages that were trimmed from the replay log. This is because if at least one other endpoint contains the message, the message is still considered spooled.

You can also choose to manually trim the replay log using PubSub+ Broker Manager or the trim-logged-messages command in addition to automatic trimming as part of your trimming strategy.

Because trimming occurs in batches and is triggered based on factors such as the size of the replay log (configured quota), and the number and size of the messages, trimming may not immediately occur when the replay log is over 90% of its configured quota for max-spool-usage. For example, if the configured quota for the replay log is very small (for example, less than 1000 MB), the target of 90% may not be perfectly maintained.

For more information about use cases for manually trimming the replay log, see:

For instructions to trim the replay log, see one of the following topics:

Impacts of Slow Trimming

If an event broker receives messages at a faster rate than it can trim the replay log, the spool usage for the replay log can increase to over 90% of its configured quota for max-spool-usage. If the spool usage reaches 100% of the configured quota, the processing of subsequent ingress messages is delayed and the event broker back-pressures publishing clients to ensure the replay log size does not exceed its configured quota.

Solace recommends that you monitor the replay log usage and choose a replay log trimming strategy. For example, if you notice that a replay log consistently exceeds 90% of its configured quota for max-spool-usage, you may need to manually trim the replay log or be more selective with the messages that are added to the message replay log. For example, you can use topic filters to reduce the number of messages written to the replay log. For more information, see Determining a Log Trimming Strategy.

For more information about monitoring and using topic filters, see:

Determining a Log Trimming Strategy

When you configure message replay, Solace recommends that you decide on a strategy to handle replay log trimming. You can simply choose to use automatic trimming where no additional configuration is required, but you should be aware that trimming the replay log may cause potential performance impacts to the event broker regarding the rate of received messages it is able to process. 

You can choose to manually trim the replay log before the 90% threshold is reached. This strategy can reduce the performance implications of automatic trimming at unexpected times. Though manual trimming has the same performance impact on the event broker, you can choose to run it at designated times when the event broker isn't sensitive to performance variations. A use case that is well-suited for manual trimming and avoids automatic trimming includes these characteristics:

  • The traffic pattern of the event broker follows a predictable cycle, in which there is a quiet time period where trimming the replay log does not cause performance impacts.

  • The replay log can be sized large enough to hold all messages logged for the minimum replay log retention that you require, plus the messages logged within the duration of a traffic cycle.

  • It's possible to deploy an automated application to perform trimming that runs on an appropriate schedule.

The following example shows how you could implement a manual trimming strategy. Consider a deployment where:

  • A system writes up to 125,000 MB of messages to the replay log each day.
  • The event broker has low traffic during nighttime, when manual replay log trimming can be performed without impacting the event broker performance.
  • The required log retention is seven days and includes an additional cycle day.

With the parameters mentioned above, you can use the following configuration:

  1. The replay log would require 1,000,000 MB without automatic trimming occurring. This sample calculation is as follows:

    • The 125,000 MB /day x (7 days of retention with an additional one day cycle) = 125,000 MB/day x 8 days = 1,000,000 MB.

    • Because trimming begins at 90% of the configured quota, you would set the max-spool-usage to 25% more than the size of the data to retain (1,250,000 MB). Because the expected spool usage won't exceed 1,000,000 MB, automatic trimming wouldn't begin until the replay log had approximately 1,125,000 MB of data and would ensure that your replay log is sufficiently sized. For more information, see Configuring the Replay Log Size.

      This design uses 80% of the replay log's configured quota for max-spool-usage, but because trimming doesn't occur until 90%, it minimizes automatic trimming from occurring and it provides you the flexibility to change the system behavior without losing important data.To ensure this design operates within the design parameters, monitoring is required, as described in step 3.

  2. In your deployment, your automated management application runs nightly to manually trim the replay log during your quiet period using one of the following mechanisms:

    • Running the enable admin message-spool message-vpn <vpn-name> replay-log trim-logged-messages <older-than-date> command where <older-than-date> is in the format YYYY-MM-DDThh:mm:ssTZD and TZD is the time zone designator. For example:

      solace1(admin/message-spool/replay-log)# trim-logged-messages older-than-date 2018-08-13T11:34:13-04:00

      For more information, see Trimming Logged Messages.

      You can also trim the replay log manually in PubSub+ Broker Manager. For more information, see Trimming the Replay Log.

    • Using the SEMPv2 command, PUT /msgVpns/{msgVpnName}/replayLogs/{replayLogs/{replayLogName}/trimLoggedMsgs (see the SEMP reference for more information).

  3. As you monitor the replay log usage, ensure the replay log usage never exceeds 80%, as described in step 1. If it does, the presumptions of the design of your strategy are no longer valid and you must re-characterize your system, adjust the replay log's configured quota for max-spool-usage, and then update the system.