Externalizing the Message Spool & Diagnostics

In production deployments, it is recommended that a Solace PubSub+ software message broker's message spool and diagnostics volumes be assigned to an external block device. In this section we show you the steps to do that assignment.

Cloud & Machine Image Procedure

In this section we go through the configuration steps you need to perform to assign the message spool and diagnostics volumes to an external block device. It's a 3-step procedure:

The last step in the procedure is to increase the message spool usage limit, max-spool-usage, on the software message broker so it can make use of the increased space for the message spool. It is recommended that you calculate max-spool-usage prior to starting the above configuration procedure, and how to do that is discussed in the next section, How to calculate max-spool-usage.

How to calculate max-spool-usage

The minimum recommended size of the diagnostics volume varies with the number of client connections to be supported, as is shown in the following table.

Minimum Recommended Diagnostic Volume Size for various Client Connection Limits

Client Connection Limit Minimum Recommended Diagnostics Volume Size (GB)
100 2.2
1,000 8.6
10,000 10.7
100,000 21.5
200,000 21.5

If the message spool and diagnostics volumes are going to share the same external block device, then the minimum size of the message spool volume is simply,

min_recommended_message_spool_volume_size = block_device_size - min_recommended_diagnostics_volume_size

The configuration parameter max-spool-usage specifies how much storage is available for the messages in the message spool. It's recommended that 20% of the message spool be reserved as overhead, so the minimum recommended value for max-spool-usage is,

max-spool-usage = 0.80 * ( block_device_size - min_recommended_diagnostics_volume_size )

As an example, consider a 30 GB external block device, the recommended values of max-spool-usage for the various client connection limits, calculated with the above equation, are shown in the following table.

Example: Minimum recommended values for max-spool-usage given a 30 GB external block device

Client Connection Limit Minimum Recommended Diagnostics Volume Size (GB) Minimum Recommended max-spool-usage (GB)
100 2.2 22.2
1,000 8.6 17.1
10,000 10.7 15.4
100,000 21.5 6.8
200,000 21.5 6.8
Notice  

When software message broker storage volumes share the same external block device, as they are assumed to be in the above discussion, it should be noted that there are no software message broker facilities to prevent the storage from being over provisioned, so it is recommended that there be no storage over provisioning. So, when multiple storage volumes are mounted on a single external block device, as were the message spool and diagnostics in the above discussion, the external block device should be at least as large as the sum of the recommended storage volume sizes.

Step 1: Configure an External Block Device

The first task is to attach an external block device, which is basically a second disk, to the software message broker virtual machine.

For best performance and maximum data integrity, especially in production deployments, itʼs recommended that you use an enterprise-grade SSD, or equivalent, that supports power loss data protection features.

The exact procedure for adding an external block device to the software message broker virtual machine depends on the hypervisor or cloud service you're using, but the following is a list of steps that are commonly encountered when performing this task:

  1. Power down the message broker if it is powered on.
  2. Using the hypervisor or cloud configuration tools, add a hardware device or storage to the virtual machine, specifying a new or existing virtual disk.
    The following are guidelines for settings that may be available in your hypervisor when creating a virtual disk:
    • The new disk must be sized in light of the considerations noted in How to calculate max-spool-usage.
    • Use a permanent disk (whereby changes are immediately and permanently written to disk).
    • Do not make the disk independent from the main disk for snapshots.
    • Use a SCSI disk if possible. For a SCSI virtual disk the virtual device node may be SCSI(0:1).
    • Use a fixed size disk.
    • Use thick, eager-zeroed disk provisioning.
    • Do not split files.
  3. Power on the message broker.
  4. Log into the message broker Linux host environment as user sysadmin. It is accessible by ssh to port 22 (port 2222 for software message broker releases prior to 8.5.0) or through the console .
  5. Confirm that a new disk device is present. It should be called /dev/sdb, /dev/vdb, /dev/xvdb, or something similar.

Step 2: Assign the Message Spool and Diagnostics to the External Block Device

To assign a software message broker's message spool and diagnostic volumes to the newly attached external block device, perform the following steps on the Linux host:

  1. Review the available external block devices, partitions and storage volumes.

    In this example, internalSpool, which is the name of the message spool, initially resides on the path /var/lib/docker/volumes/internalSpool/_data and diagnostics resides on /var/lib/docker/volumes/diagnostics/_data. Also, there is an external block device called vdb to which internalSpool and diagnostics will be assigned.

     >sudo solacectl storage ls
    Block Devices:
    Name Size Note
    vda 30.0G Main device
    └─vda1 0.2G
    └─vda2 29.8G
    vdb 30.0G

    Storage Volumes:
    Name Size Used Available Path
    /dev/mapper/vg01-root 9.8G 1.1G 8.8G /
    /dev/mapper/vg01-solace 9.8G 1.4G 8.5G /var/lib/docker/volumes
    ├─adb 9.8G 1.0G 8.5G /var/lib/docker/volumes/adb/_data
    ├─adbBackup 9.8G 0 8.5G /var/lib/docker/volumes/adbBackup/_data
    ├─diagnostics 9.8G 4.0K 8.5G /var/lib/docker/volumes/diagnostics/_data
    ├─etc 9.8G 8.0K 8.5G /etc/solace/solace-container.d
    ├─internalSpool 9.8G 274M 8.5G /var/lib/docker/volumes/internalSpool/_data
    ├─jail 9.8G 9.8M 8.5G /var/lib/docker/volumes/jail/_data
    └─var 9.8G 25M 8.5G /var/lib/docker/volumes/var/_data

    Image Pool:
    Name Size Used Available
    Image pool 8.905 GB 1.033 GB 7.872 GB

  2. Stop the message broker if it's running.

    >solacectl service stop

  3. Assign internalSpool and diagnostics to vdb.

    >sudo solacectl storage configure vdb internalSpool diagnostics
    Creating new partition...
    Verified the file system is XFS
    Mounting /dev/vdb1
    Volumes have been configured in the new block device.

  4. Review to confirm the assignment of internalSpool and diagnostics to vdb.

    >sudo solacectl storage ls
    Block Devices:
    Name Size Note
    vda 30.0G Main device
    └─vda1 0.2G
    └─vda2 29.8G
    vdb 30.0G
    └─vdb1 30.0G

    Storage Volumes:
    Name Size Used Available Path
    /dev/mapper/vg01-root 9.8G 1.1G 8.8G /
    /dev/mapper/vg01-solace 9.8G 1.1G 8.7G /var/lib/docker/volumes
    ├─adb 9.8G 1.0G 8.7G /var/lib/docker/volumes/adb/_data
    ├─adbBackup 9.8G 0 8.7G /var/lib/docker/volumes/adbBackup/_data
    ├─etc 9.8G 8.0K 8.7G /etc/solace/solace-container.d
    ├─jail 9.8G 9.8M 8.7G /var/lib/docker/volumes/jail/_data
    └─var 9.8G 25M 8.7G /var/lib/docker/volumes/var/_data
    /dev/vdb1 30G 306M 30G /mnt/vmr
    ├─diagnostics 30G 4.0K 30G /mnt/vmr/diagnostics
    └─internalSpool 30G 274M 30G /mnt/vmr/internalSpool

    Image Pool:
    Name Size Used Available
    Image pool 8.905 GB 1.032 GB 7.872 GB
  5. Restart the software message broker

    >solacectl service start

To undo the above assignment, the message spool internalSpool can be re-assigned to the main device, vda, with the command: sudo solacectl storage configure vda internalSpool. Likewise, diagnostics can be re-assigned to the main device with: sudo solacectl storage configure vda diagnostics.

Step 3: Increase Message Spool Usage

After the message spool has been migrated to the external block device, you must change the configured maximum spool usage limit, max-spool-usage, to access the increased capacity, with the following Solace CLI command:

solace(configure)# hardware message-spool max-spool-usage <new-maximum>

Docker Container Image Procedure

To provide sufficient storage space for the message spool and diagnostics volumes, it is recommended to assign them to an external block device. For instructions, see Assigning External Storage.