Defragmenting the Guaranteed Messaging Spool

The spool files that an event broker uses to store Guaranteed messages may become fragmented over time if consumers go offline and do not reconnect. Spool files cannot be deleted until all of the messages in the file have been delivered and ACKed by all consumers. Fragmentation occurs when a small number of undeliverable messages are retained for offline consumers, which prevents delivered messages in the same spool file from being deleted. Fragmentation can consume large quantities of disk space with messages that have already been delivered to consumers. Defragmentation moves the undelivered messages to a new spool file, allowing the delivered messages in the spool file to be deleted.

You can either manually trigger defragmentation (see Manual Defragmentation ), or configure the event broker to automatically defragment the message spool (see Automatic Defragmentation).

In general:

  • All manual and automatic (schedule-based and threshold-based rules) defragmentation behaviors can coexist and function together.
  • Any automatic defragmentation operation is skipped if a defragmentation operation is already in progress.
  • All manual and automatic defragmentation operations can be manually stopped by the administrator.

Manual Defragmentation

To manually trigger defragmentation, enter the following command. When started, the message spool file defragmentation runs in the background. 

solace(admin)# system message-spool 
solace(admin/system/message-spool)# defragment-spool-files start

To stop an active spool file defragmentation process, enter the following command:

solace(admin/system/message-spool)# defragment-spool-files stop
  • Defragmentation may not be able to completely defragment the message spool if there are spooled messages associated with open transactions. Defragmentation starts with the oldest message spooled and defragments the messages in the order they arrived until it reaches either the newest message or a message that is part of an open transaction. Information about the open transaction for the most recent defragmentation is available in the Exit Condition field when you issue the show message-spool command. Once the transactions are committed or rolled back defragmentation can be run again. In some cases committing or rolling back a transaction may not be possible if the transaction is not in a prepared state. In these cases, a client disconnect may be required to resolve the blocked transactions.
  • When an HA redundant pair of appliance event brokers is used, message spool defragmentation can only be performed on the active appliance event broker. If an HA failover occurs, the message spool defragmentation stops. To restart the defragmentation process, you must make the failed appliance event broker active again, then execute the defragment-spool-files start command again. To defragment the message spool on the backup appliance event broker, initiate an HA failover to make the backup appliance event broker active. When defragmentation is initiated on an active software event broker, the message spool on both the active and standby are defragmented.
  • If the Active Disk Partition Usage shows 100% in the show message-spool detail command output, defragmentation cannot free up disk space. You must first consume messages to reduce disk usage below 100% before defragmentation can work effectively.

CLI examples of how to manually start defragmenting a spool file, as well as check its status, and stop it from running are shown below:

solace(admin)# system message-spool
solace(admin/system/message-spool)# defragment-spool-files start
Defragmentation has been initiated.

solace# show message-spool detail
 
Config Status:                            Enabled (Primary)
Maximum Spool Usage:                      60000 MB

. . .
 
Operational Status:                       AD-Active
Datapath Status:                          Up
Synchronization Status:                   Synced
 
. . .    
            
Defragmentation:
 Status:                                 Active (8.8%)
  Schedule Enabled:                       No
    Days:                                 daily
    Times:                                0:00
  Threshold Enabled:                      No
    Fragmentation:                        50% 
    Spool Usage:                          50%
    Minimum Interval:                     15 minutes
  Estimated Fragmentation:                12%
  Estimated Recoverable Space:            123 MB  
  Last Result:
    Completed On:                         May 10 2020 10:11:12
    Completion %:                         100%
    Exit Condition:                       Success
Number of delete in-progress:             0

. . .
 
                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000       0.0000          0.0000
Number of Messages Currently Spooled            0            0               0

. . .
solace(admin/system/message-spool)# defragment-spool-files stop    
Defragmentation has been stopped.
solace# show message-spool detail
 
Config Status:                            Enabled (Primary)
Maximum Spool Usage:                      60000 MB

. . .
 
Operational Status:                       AD-Active
Datapath Status:                          Up
Synchronization Status:                   Synced
 
. . .
            
Defragmentation:
 Status:                                 Idle
  Schedule Enabled:                       No
    Days:                                 daily
    Times:                                0:00
  Threshold Enabled:                      No
    Fragmentation:                        50% 
    Spool Usage:                          50%
    Minimum Interval:                     15 minutes
  Estimated Fragmentation:                12%
  Estimated Recoverable Space:            123 MB
  Last Result:
    Completed On:                         May 10 2020 10:11:12
    Completion %:                         58%
    Exit Condition:                       Unmovable XA Transaction       
                                          XID:XXX-XXXXXXX
Number of delete in-progress:             0

. . .
 
                                              ADB         Disk           Total
Current Persistent Store Usage (MB)        0.0000       0.0000          0.0000
Number of Messages Currently Spooled            0            0               0

. . .

Automatic Defragmentation

You can configure an event broker to defragment the message spool automatically, triggered in one of two ways:

You can configure an event broker to perform both scheduled and threshold-based defragmentation, the two options aren’t mutually exclusive.

Schedule-Based Defragmentation

To schedule the days of the week to trigger defragmentation, enter the following commands:

solace(config)# hardware 
solace(config/hardware)# message-spool
solace(config/hardware/message-spool)# defragment-spool-files
solace(.../message-spool/defragment-spool-files)# schedule
solace(...spool/defragment-spool-files/schedule)# days <days-of-week>

Where:

<days-of-week> is either daily or a comma-separated list of days or numbers where 0 is Sunday, 1 is Monday, etc.

The no version of this command, no days, returns the value to the default, daily.

To schedule times per day to trigger defragmentation, enter the following command:

solace(...spool/defragment-spool-files/schedule)# times <times-of-day>

Where:

<times-of-day> is either hourly or a comma-separated list of up to four times of the form hh:mm where hh is between 0 and 23, and mm is between 0 and 59.

The no version of this command, no times, returns the value to the default, 0:00.

By default, schedule-based defragmentation is disabled.To enable scheduled-based defragmentation, enter the following command:

solace(...spool/defragment-spool-files/schedule)# no shutdown

To disable scheduled-based defragmentation, enter the following command:

solace(...spool/defragment-spool-files/schedule)# shutdown

Threshold-Based Defragmentation

Once enabled, threshold-based defragmentation operations are triggered by all of the following three criteria being true:

  • The amount of fragmentation has reached a certain percent.
  • The amount of spool usage has reached a certain percent.
  • A minimum amount of time has passed since the previous defragmentation (regardless of its trigger).

To configure the percentage of fragmentation that triggers defragmentation, enter the following commands:

solace(config)# hardware 
solace(config/hardware)# message-spool
solace(config/hardware/message-spool)# defragment-spool-files
solace(.../message-spool/defragment-spool-files)# threshold
solace(...pool/defragment-spool-files/threshold)# fragmentation-percentage <percentage>

Where:

<percentage> is the percentage of fragmentation between 30 and 100.

The no version of this command no fragmentation-percentage returns the value to the default, 50 percent.

To configure the percentage of spool usage that triggers defragmentation, enter the following command:

solace(...pool/defragment-spool-files/threshold)# usage-percentage <percentage>

Where:

<percentage> is the percentage of spool usage between 30 and 100.

no version of this command, no usage-percentage, returns the value to the default, 50 percent. specifies the minimum amount of time

To configure the minimum amount of time that must have elapsed since the previous defragmentation, enter the following command:

solace(...pool/defragment-spool-files/threshold)# min-interval <interval>

Where:

<interval> the minimum interval in minutes.

no version of this command, no min-interval, returns the value to the default, 15 minutes.

By default, threshold-based defragmentation is disabled. To enable threshold-based defragmentation, enter the following command:

solace(...pool/defragment-spool-files/threshold)# no shutdown

To disable threshold-based defragmentation, enter the following command:

solace(...pool/defragment-spool-files/threshold)# shutdown