Gathering Diagnostics from Software Event Broker Containers

A complete set of event broker diagnostic files and debug logs are sometimes required for Solace to determine the root cause of an operational problem. To assist with the investigation, you can use either of the following to gather diagnostic information about a container that runs the PubSub+ software event broker:

  • gather-diagnostics-host utility from the container
  • gather-diagnostics command from the Solace CLI

Once diagnostics are gathered, core dump files are deleted from the software event broker container to save space. The gathered diagnostic logs are encrypted by default when you run the gather-diagnostics-host utility or the gather-diagnostics command.

Using the Gather-Diagnostics-Host Utility

We recommend that you use the gather-diagnostics-host utility instead of the gather-diagnostics command in the Solace CLI. The utility can be run even if the container for the software event broker isn’t running, Solace CLI is not accessible, or if there is a suspected misconfiguration of the container host.

To use the gather-diagnostics-host utility, you must have a Linux host with the a version of the Python SDK that is in active support and has security update support. For more information, see the Python Version Support site.

The commands below assume that you are running your container in rootful mode. If you are running in rootless mode, you may not need to use sudo.

The gather-diagnostics-host utility is delivered in the software event broker container file system. To extract the utility, use the docker cp or podman cp command.

>sudo docker cp <CONTAINER_NAME>:/usr/share/solace/gather-diagnostics-host <DESTINATION_PATH>

For example:

>sudo docker cp solace:/usr/share/solace/gather-diagnostics-host /usr/local/bin

The gather-diagnostics-host utility attempts to gather core dump files from:

  1. Any container host path mounted as a volume in the software event broker container at /var/lib/solace/diags.
  2. The container host path specified in the Linux kernel core_pattern if that path is specified as an absolute path. That is, relative paths and pipes to user space programs will not be interpreted. For more information, see Managing Core Files.
  3. A user-specified path provided as a gather-diagnostics-host command line parameter.

gather-diagnostics-host collects diagnostics information from software event broker containers. When run without additional command line parameters, it will gather information from all containers created from the default container image named solapp:currentload or, if none exist, containers using the default name solace. The user may specify a container, by name, using --name, or a specific container image, using --image.

To run the gather-diagnostics-host utility, enter the following command in the Linux host environment:

gather-diagnostics-host [-h] [-f <filename>] [-d <days>] [-n <container>] [-i <repository>:<tag>] [--no-container] [--core-host-path pattern] [-V] [-D] [--no-encrypt]

Where:

-h or --help displays the gather diagnostics help information.

-f filename or --file filename outputs a single compressed file of diagnostics data. A default file name is used if one is not specified.

-d days or --days days specifies the number of days of history to capture.

-n container or --name container specifies a particular container name or ID to gather diagnostic data for. If no container is specified, all containers matching the container image are used.

-i repository:tag or --image repository:tag specifies the container image to use to find matching containers. The default value is solace-app:currentload.

--no-container specifies do not attempt to run gather-diagnostics-host inside the container.

--core-host-path pattern specifies the path to core files to be used as regex. This script captures all files on the host matching pattern.

-V specifies to increase output verbosity.

-D specifies debug mode, which describes utility actions without doing anything.

--no-encrypt returns the data in a non encrypted file ending in .tgz . By default, files are encrypted and end in .tgz.p7m.

Once diagnostics are gathered, core dump files are deleted from the software event broker container to save space. You can use --no-encrypt to view the gathered logs before sending them. We recommend that you do not send any log files without encryption.

To encrypt a file previously generated with --no-encrypt, or another file you want to send, use the encrypt command that is delivered in the container file system for the software event broker. To extract the command, use the following docker cp or podman cp command:

>sudo docker cp <CONTAINER_NAME>:/usr/sw/loads/currentload/scripts/encrypt <DESTINATION_PATH>

For example:

>sudo docker cp solace:/usr/sw/loads/currentload/scripts/encrypt /usr/local/bin

To run the encrypt command, enter the following command in the Linux host environment:

encrypt [-h] [-V] <input_file> <output_file>

Where:

-h or --help displays the encrypt help information.

-V specifies to increase output verbosity.

<input file> specifies the file to encrypt.

<output file> specifies the name of the encrypted file which must have a .p7m extension.

Using Gather-Diagnostics-Host on Docker for Windows

For convenience, the gather-diagnostics-host support utility is bundled in the software event broker container image. To use this utility in Docker for Windows, create a new sidecar container with privileged settings based on the software event broker container image with a different ENTRYPOINT command (gather-diagnostics-host).

For example: 

> docker run --rm -t --privileged -u root --volumes-from solace -v c:\Users\username\storage\diags:/tmp -v /var/run/docker.sock:/var/run/docker.sock solace-pubsub-standard:<version> /usr/share/solace/gather-diagnostics-host --sidecar

This example creates a sidecar container which runs the gather-diagnostic-host utility on the running container named solace and places the output in a compressed tar file in the C:\Users\username\storage\diags directory.

By default, the gather-diagnostics-host utility places the /tmp directory inside the new container and removes it when the container is removed. Solace recommends mounting the /tmp directory to a persistent location on a shared disk as demonstrated in the example above.

Using Gather-Diagnostics-Host Utility on Docker for macOS

For convenience, the gather-diagnostics-host support utility is bundled in the software event broker container image. To use this utility in Docker for macOS, create a new sidecar container with privileged settings based on the container image with a different ENTRYPOINT command (gather-diagnostics-host).

For example:

> docker run --rm -t --privileged -u root --volumes-from solace -v /Users/username/storage/diags:/tmp -v /var/run/docker.sock:/var/run/docker.sock solace-pubsub-standard:<version> /usr/share/solace/gather-diagnostics-host --sidecar

This example creates a sidecar container which runs the gather-diagnostic-host utility on the running container named solace and places the output in a compressed tar file in the Users/username/storage/diags directory.

By default, the gather-diagnostics-host utility places the /tmp directory inside the new container and removes it when the container is removed. Solace recommends mounting the /tmp directory to a persistent location on a shared disk as demonstrated in the previous example.

Using the Gather-Diagnostics Solace CLI Command

In the Solace CLI, you can use the gather-diagnostics command to gather diagnostics for a software event broker container if you don't have access to the container host; that is, if you only have Solace CLI access to the container for the software event broker.

The gather diagnostics command automatically gathers and packages the information. When the command completes, the retrieved files and logs are saved as a single compressed file, and the name of this diagnostics file and its location are printed to the screen.

A File Transfer User Account is required to access the generated diagnostic file. For more information about account creation see Configuring File Transfer User Accounts

You can use third-party applications, such as Filezilla or WinSCP (Windows), or command-line utilities, such as SCP or FTP, to access the compressed files to transfer the files between the event broker and another location, such as your computer. To do this, enter the username and password of the File Transfer User Account as credentials in the third-party application to access the event broker.

To gather event broker diagnostics, enter the following commands:

solace> enable 
solace# admin
solace(admin)# gather-diagnostics [days-of-history <days>]

Where:

days-of-history <days> specifies the number of days to gather diagnostics data for. By default, diagnostic data is gathered for one day.

no-encrypt returns the data in a non encrypted file that ends in .tgz.

By default, files are encrypted and end in .tgz.p7m.

Once diagnostics are gathered, core dump files are deleted from the software event broker container to save space. You can use no-encrypt to view the gathered logs before sending them. We recommend that you do not send any log files without encryption.