Defragmenting the Guaranteed Messaging Spool

The spool files that a Solace PubSub+ event broker uses to store Guaranteed messages may become "fragmented" over time when consumers frequently go offline and do not reconnect. Fragmentation can occur because the small number of messages awaiting delivery to those offline consumers are maintained, which prevents the larger number of messages that have been consumed that are also on the spool file from being removed. This has the unwanted effect of consuming large quantities of disk space with messages that have already been delivered to consumers.

To remove the delivered messages from spool files and reclaim disk space, you can either manually trigger defragmentation (see Manual Defragmentation ) or configure the event broker to automatically defragment the message spool (see Automatic Defragmentation).

In general:

All manual and automatic (schedule-based and threshold-based rules) defragmentation behaviours can coexist and function together.
Any automatic defragmentation run (automatic or manual) is ignored (skipped) if a defragmentation run is already in progress (regardless of what triggered it).
All manual and automatic defragmentation runs can be manually stopped by the administrator.

Manual Defragmentation

To manually trigger defragmentation, enter the following command. When started, the message spool file defragmentation will run in the background.

solace(admin)# system message-spool 
solace(admin/system/message-spool)# defragment-spool-files start

To stop an active spool file defragmentation process, enter the following command:

solace(admin/system/message-spool)# defragment-spool-files stop

Defragmentation may not be able to completely defragment the message spool if there are spooled messages associated with open transactions. Defragmentation starts with the oldest message spooled and defragments the messages in the order they arrived until it reaches either the newest message or a message that is part of an open transaction. Information about the open transaction for the most recent defragmentation is available in the Exit Condition field when you issue the show message-spool command. Once the transactions are committed or rolled back defragmentation can be run again. In some cases committing or rolling back a transaction may not be possible if the transaction is not in a prepared state. In these cases, a client disconnect may be required to resolve the blocked transactions.
When an HA redundant pair of appliances is used, message spool defragmentation can only be performed on the active appliance. If an HA failover occurs, the message spool defragmentation will stop. To restart the defragmentation process, you must make the failed appliance active again, then execute the defragment-spool-files start command again. To defragment the message spool on the backup appliance, initiate an HA failover to make the backup appliance active. This only applies to appliances. When defragmentation is initiated on an active software event broker, the message spool on both the active and standby are defragmented.

CLI examples of how to manually start defragmenting a spool file, as well as check its status, and stop it from running are shown below:

solace(admin)# system message-spool
solace(admin/system/message-spool)# defragment-spool-files start
Defragmentation has been initiated.

solace# show message-spool detail
 
Config Status:                            Enabled (Primary)
Maximum Spool Usage:                      60000 MB

. . .
 
Operational Status:                       AD-Active
Datapath Status:                          Up
Synchronization Status:                   Synced
 
. . .	
			
Defragmentation:
 Status:                                 Active (8.8%)
  Schedule Enabled:                       No
    Days:                                 daily
    Times:                                0:00
  Threshold Enabled:                      No
    Fragmentation:                        50% 
    Spool Usage:                          50%
    Minimum Interval:                     15 minutes
  Estimated Fragmentation:                12%
  Estimated Recoverable Space:            123 MB  
  Last Result:
    Completed On:                         May 10 2020 10:11:12
    Completion %:                         100%
    Exit Condition:                       Success
Number of delete in-progress:             0

. . .
 
                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000       0.0000          0.0000
Number of Messages Currently Spooled            0            0               0

. . .

solace(admin/system/message-spool)# defragment-spool-files stop	
Defragmentation has been stopped.

solace# show message-spool detail
 
Config Status:                            Enabled (Primary)
Maximum Spool Usage:                      60000 MB

. . .
 
Operational Status:                       AD-Active
Datapath Status:                          Up
Synchronization Status:                   Synced
 
. . .
			
Defragmentation:
 Status:                                 Idle
  Schedule Enabled:                       No
    Days:                                 daily
    Times:                                0:00
  Threshold Enabled:                      No
    Fragmentation:                        50% 
    Spool Usage:                          50%
    Minimum Interval:                     15 minutes
  Estimated Fragmentation:                12%
  Estimated Recoverable Space:            123 MB
  Last Result:
    Completed On:                         May 10 2020 10:11:12
    Completion %:                         58%
    Exit Condition:                       Unmovable XA Transaction       
                                          XID:XXX-XXXXXXX
Number of delete in-progress:             0

. . .
 
                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000       0.0000          0.0000
Number of Messages Currently Spooled            0            0               0

. . .

Automatic Defragmentation

You can configure an event broker to defragment the message spool automatically, triggered in one of two ways:

Based on a fixed time schedule, for example every day at 1 am (see Schedule-Based Defragmentation).
Based on a threshold of fragmentation and usage being exceeded (see Threshold-Based Defragmentation).

You can configure an event broker to perform both scheduled and threshold-based defragmentation, the two options aren’t mutually exclusive.

Schedule-Based Defragmentation

To schedule the days of the week to trigger a defragmentation run, enter the following commands:

solace(config)# hardware 
solace(config/hardware)# message-spool
solace(config/hardware/message-spool)# defragment-spool-files
solace(.../message-spool/defragment-spool-files)# schedule
solace(...spool/defragment-spool-files/schedule)# days <days-of-week>

Where:

<days-of-week> is either daily or a comma-separated list of days or numbers where 0 is Sunday, 1 is Monday, etc.

The no version of this command, no days, returns the value to the default, daily.

To schedule times per day to trigger a defragmentation run, enter the following command:

solace(...spool/defragment-spool-files/schedule)# times <times-of-day>

Where:

<times-of-day> is either hourly or a comma-separated list of up to four times of the form hh:mm where hh is between 0 and 23, and mm is between 0 and 59.

The no version of this command, no times, returns the value to the default, 0:00.

By default, schedule-based defragmentation is disabled.To enable scheduled-based defragmentation, enter the following command:

solace(...spool/defragment-spool-files/schedule)# no shutdown

To disable scheduled-based defragmentation, enter the following command:

solace(...spool/defragment-spool-files/schedule)# shutdown

Threshold-Based Defragmentation

Once enabled, threshold-based defragmentation runs are triggered by all of the following three criteria being true:

The amount of fragmentation has reached a certain percent.
The amount of spool usage has reached a certain percent.
A minimum amount of time has passed since the previous defragmentation run (regardless of its trigger).

To configure the percentage of fragmentation that will trigger a defragmentation run, enter the following commands:

solace(config)# hardware 
solace(config/hardware)# message-spool
solace(config/hardware/message-spool)# defragment-spool-files
solace(.../message-spool/defragment-spool-files)# threshold
solace(...pool/defragment-spool-files/threshold)# fragmentation-percentage <percentage>

Where:

<percentage> is the percentage of fragmentation between 30 and 100.

The no version of this command no fragmentation-percentage returns the value to the default, 50 percent.

To configure the percentage of spool usage that will trigger a defragmentation run, enter the following command:

solace(...pool/defragment-spool-files/threshold)# usage-percentage <percentage>

Where:

<percentage> is the percentage of spool usage between 30 and 100.

no version of this command, no usage-percentage, returns the value to the default, 50 percent. specifies the minimum amount of time

To configure the minimum amount of time that must have elapsed since any previous defragmentation run, enter the following command:

solace(...pool/defragment-spool-files/threshold)# min-interval <interval>

Where:

<interval> the minimum interval in minutes.

no version of this command, no min-interval, returns the value to the default, 15 minutes.

By default, threshold-based defragmentation is disabled. To enable threshold-based defragmentation, enter the following command:

solace(...pool/defragment-spool-files/threshold)# no shutdown

To disable threshold-based defragmentation, enter the following command:

solace(...pool/defragment-spool-files/threshold)# shutdown

Provide feedback