PubSub+ Insights for Container-Based Software Event Brokers

You can benefit from the observability features of PubSub+ Insights for your self-managed PubSub+ software event brokers by installing an Insights Agent in your container environment. You can deploy the Insights Agent to both Docker and Podman Container Engines.

After installing and configuring the Insights Agent you can access and use the numerous Insights dashboards, monitors, and metrics provided by Solace in your Datadog sub-organization. For more information, see PubSub+ Insights Advanced Monitoring.

Before deploying Insights for your Container-based software event broker, review the Considerations for Deploying Insights for Container-Based Software Event Brokers and the Insights Agent Resource Requirements.

The process for installing the Insights Agent to your container-based environment includes in three steps:

1. Downloading the Insights Agent

2. Configuring and Installing the Insights Agent

3. Validating the Insights Agent Installation

After installing the Insights Agent, you must periodically update it when new versions become available. Failure to update the Insights Agent may affect its ability to ingest your software event broker metrics and log and provide them to Datadog. For more information, see Upgrading the Insights Agent.

You can uninstall the Insights Agent if required. For more information, see Uninstalling the Insights Agent.

Considerations for Deploying Insights for Container-Based Software Event Brokers

Be aware of the following considerations when using Insights with a self-managed software event broker in Docker or Podman:

• Review the Considerations for Using Insights with Self-Managed Software Event Brokers.

• You must use the Add Insights Monitoring form in the Cloud Console to acquire the image pull secret and variable value referenced in the CLI commands in the procedures in this document.

• Solace recommends securely saving the commands used in the procedures below on a per-installation basis. You will require them in the future when upgrading the Insights Agent.

• The CLI commands provided throughout this document use 'docker', however, they are identical for both Docker and Podman, unless otherwise noted in the procedures. If installing the Insights Agent to a Podman, replace 'docker' in our CLI examples with 'podman'.

• Your Container Engine must meet these minimum version requirements:

  • Docker version 26.0.0 or greater

  • Podman version 5.4.0 or greater

Insights Agent Resource Requirements

Solace recommends providing each Insights Agent with the following minimum resources when deploying in Docker or Podman:

Downloading the Insights Agent

You have two choices for downloading the Insights Agent:

• Pulling the Insights Agent Image from the Solace Container Registry

• Downloading the Insights Agent from Products

Pulling the Insights Agent Image from the Solace Container Registry

You can configure your deployment to pull the image directly from the Solace Container Registry. You must use the registry secret provided by the Cloud Console (config.json) after completing the Add Insights Monitoring form to pull the image from the Solace Container Registry.

Save the config.json file to a secure location on your Docker host, for example /path/to/config/json.

Use the above path where you saved the config.json file with the --config flag in your docker commands that require pulling the image.

Alternatively, if you don't want to pull images using the docker --config command, you can use the credentials in the config.json file to manually perform a docker login to the Solace Container Registry. Doing so merges the authentication credentials with your existing Docker configuration.

Downloading the Insights Agent from Products

You can download the Insights Agent image tar files from products.solace.com. If you require assistance, contact Solace.

Configuring and Installing the Insights Agent

You will need to reuse the configuration files and docker run commands used in the following sections in the future when you upgrade the Insights Agent. Solace recommends securely saving these configuration files and commands on a per-installation basis, in a way that identifies the installation by container name so you can reuse them.

After downloading and saving the Insights Agent registry secret, you can install the Insights Agent container using the CLI commands below. We provide two examples:

• Using an Environment File to Run the Insights Agent

• Using Line-by-Line Variable Specification to Run the Insights Agent

Running either of the commands produces the following results:

• Runs the Insights Agent in a container named {INSIGHTS_AGENT_CONTAINER_NAME}. The container runs:

  • in detached mode

  • with the default root user ID 0

• Injects the environment variables required to configure the Insights Agent into its container.

• Shares the software event broker's /var/lib/solace/jail/logs path with the Insights Agent under /jail/logs.

• Provides the container with recommended resources, as outlined in Insights Agent Resource Requirements.

• Pulls the latest version of the Insights Agent image from the Solace Container Registry.

You configure the Insights Agent using the environment variables generated by the Cloud Console after you compete the Add Insights Monitoring form. You can modify the rest of the commands to suit your requirements. We provide guidance for adding the environment variables and updating the commands below. For more information, see Configuring the Installation Command.

You must run the command to install an Insights Agent on every node (messaging and monitoring) in a high-availability event broker, using the environment variables specific to each node.

Using an Environment File to Run the Insights Agent

To run the Insights Agent using an environment file, you must use the event broker properties generated by the Cloud Console after completing the Add Insights Monitoring form to create an environment file. Then, run the following command:

docker --config /path/to/config/json run \
  --name {INSIGHTS_AGENT_CONTAINER_NAME}     # Descriptive container name \
  -d                                         # Run in detached mode \
  --restart=always                           # Optional: Restart policy \
  --network {SOLACE-NET}                     # Connect to the same network as the broker \
  --env-file {./PRIMARY-NODE.ENV}            # Load environment variables from file \
  # Mount broker logs (read-only) - adjust 'src' and 'volume-subpath' as needed
  --mount type=volume,src={BROKER-STORAGE},target=/jail/logs,volume-subpath=jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended
  --mount type=volume,src={MY-SERVICE-PRIMARY-INSIGHTS-AGENT-STATE},target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" --memory="512m" \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Using Line-by-Line Variable Specification to Run the Insights Agent

To run the Insights Agent using a line-by-line variable specification, you must replace the environment variables (for example, {INSIGHTS_AGENT_TAGS}) in the code snippet below with the environment variables provided by the Cloud Console as KEY=VALUE pairs after completing the Add Insights Monitoring form as -e flags in the command:

docker --config /path/to/config/json run \
  --name {INSIGHTS_AGENT_CONTAINER_NAME}   # Descriptive container name \
  # Environment variables from Cloud Console (Step 2) \
  -e INSIGHTS_AGENT_API_KEY="{INSIGHTS_AGENT_API_KEY}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_SITE="{INSIGHTS_AGENT_SITE"} provided by the Cloud Console \
  -e INSIGHTS_AGENT_TAGS="{INSIGHTS_AGENT_TAGS}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME="{INSIGHTS_AGENT_BROKER_HOSTNAME}"    # Or IP/Container Name \
  -e INSIGHTS_AGENT_NODE_ROLE="{INSIGHTS_AGENT_NODE_ROLE}" \
  -e INSIGHTS_AGENT_SEMP_USERNAME="INSIGHTS_AGENT_SEMP_USERNAME}" \
  -e INSIGHTS_AGENT_SEMP_PASSWORD="{INSIGHTS_AGENT_SEMP_PASSWORD}" \
  -e INSIGHTS_AGENT_SEMP_PROTOCOL="{INSIGHTS_AGENT_SEMP_PROTOCOL}" \
  -e INSIGHTS_AGENT_SEMP_PORT="{INSIGHTS_AGENT_PORT}" \
  -e INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION="{INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION}" \
  -e INSIGHTS_AGENT_HEALTH_CHECK_PORT="{INSIGHTS_AGENT_HEALTH_CHECK_PORT}" \
  # Add {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} if needed \
  # Mount broker logs (read-only) \
  --mount type=volume,src={BROKER-STORAGE},target=/jail/logs,volume-subpath=jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended \
  --mount type=volume,src={AGENT-CONTAINER-NAME}-state,target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" --memory="512m" \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Configuring the Installation Command

You can find guidance on configuring the environment variables and other portions of the commands above to meet the specific requirements of your environment in the sections below:

• Naming the Insights Agent Container

• Networking Configuration Requirements

• Determining the User ID for the Insights Agent

• Configuring Environment Variables

• Sharing the Log Files With the Insights Agent

• Persisting the Insights Agent’s Log Tailing State

Naming the Insights Agent Container

You must name the Insights Agent container. We recommend using the INSIGHTS_AGENT_BROKER_HOSTNAME variable generated by the Cloud Console after completing the Add Insights Monitoring form, with -insights-agent appended for clarity:

{INSIGHTS_AGENT_CONTAINER_NAME}=INSIGHTS_AGENT_BROKER_HOSTNAME

You can alternatively use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE environment variable, outlined in the Configuring Environment Variables table, to name the Insights Agent container. Using the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE may provide better identifiers for your Insights Agents in Insights.

Networking Configuration Requirements

Your Docker or Podman network must meet the following requirements:

• If the {INSIGHTS_AGENT_BROKER_HOSTNAME} environment variable is set to your software event broker's container name, then you should run the Insights Agent container on the same Docker or Podman network to allow host resolution with container names. Solace recommends this configuration as a best practice.

• The Docker or Podman network must allow the Insights Agent to query the software event brokers SEMP endpoints.

Determining the User ID for the Insights Agent

You must run the Insights Agent with a user that has at least read access to the share log files at /jail/logs. Solace recommends running the Insights Agent with the same user ID (or group ID) used to create the software event broker.

You can omit the -u argument to run the Insights Agent container with UID 0 (root). You must run the container with a UID 0 (root) if you want to persist the Insights Agent log tailing state in a Docker container.

The provided example uses user ID 1001. The User ID can be arbitrary and does not need to correspond to any existing or real user accounts. If the broker's log directory has group-based permissions, you can use a supplemental group ID, which likewise does not have to correspond to any existing or real group IDs. You can provide this supplemental GID in your Docker run command. For example, the following equivalent user run flags use a UID of 1001 and a supplemental GID of 1001:

--user 1001 --group-add 1001

Configuring Environment Variables

You must configure the environment variables using the environment variables generated by the Cloud Console after you complete the Add Insights Monitoring form.

If you are Using an Environment File to Run the Insights Agent you must create an environment file using these values.

The following table lists the environment variables, specifies if the variable is required, how the value is provided (by the Cloud Console or by you, the customer), the purpose of variable, and any additional important information.

Sharing the Log Files With the Insights Agent

You must give the Insights Agent read-only access to your event brokers log directory, which you can do by mounting the event broker's log directory into the Insights Agent's container at /jail/logs. The path to your event broker's log directory depends on your container setup, but is usually located within the event broker's main data volume; common locations include:

• /var/lib/solace/jail/logs

• /usr/sw/jail/logs

You must mount the host path or volume subpath that corresponds to this location. You have two options for configuring this requirement:

• Sharing Logs From a Named Docker or Podman Volume

• Sharing Logs From a Host Path Mount

Sharing Logs From a Named Docker or Podman Volume

--mount type=volume,src={STORAGE-GROUP},target=/jail/logs,volume-subpath=jail/logs,readonly

--mount type=bind,src=/mnt/solace/jail/logs,target=/jail/logs,readonly

Persisting the Insights Agent’s Log Tailing State

You can prevent missing software event broker logs caused by container restarts by configuring the Insights Agent to persist its log position markers in the /opt/datadog-agent/run directory.

In the following commands, {INSIGHTS-AGENT-CONTAINER-NAME} is the name of the Insights Agent container. You must repeat these commands for each Insights Agent.

1. Create a dedicated volume to allow the Insights Agent to persist its log position.

docker volume create {INSIGHTS-AGENT-CONTAINER-NAME}-state

2. Mount the volume (read-write) to the Insights Agent.

   • For Docker and Podman, when running the Insights Agent as root (UID 0)

     --mount type=volume,src={INSIGHTS-AGENT-CONTAINER-NAME}-state,target=/opt/datadog-agent/run

   • For Podman, when running as a non-root user:

     --mount type=volume,src={INSIGHTS-AGENT-CONTAINER-NAME}-state,target=/opt/datadog-agent/run,chown

If you configured Sharing Logs From a Host Path Mount, you configure a similar approach by giving the Insights Agent read and write access to the Insights Agent's state directory.

Validating the Insights Agent Installation

After configuring the Insights Agent, you can validate the Insights Agent is working by logging into your Insights Datadog account and viewing the Insights Agent host in the Datadog infrastructure list.

To view the Insights Agent host in the Datadog infrastructure list, do the following:

1. Log in to the PubSub+ Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the PubSub+ Cloud Console.
2. Select Insights on the navigation bar.
3. Click View in Datadog from any of the available tiles where it appears.
4. If prompted to do so, log in to Datadog.
5. From the menu on the left, select Infrastructure and then click Hosts.
6. In the Search by field, enter one of the following values to filter the list to show only your Insights Agent host:
   • {INSIGHTS_AGENT_BROKER_HOSTNAME} where {INSIGHTS_AGENT_BROKER_HOSTNAME} is the name of the Insights Agent.
   • {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} where {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} is the name of the Insights Agent if you set this environment variable.
7. Select the host to access its information. From this screen you can:
   1. click the Host Info tab to confirm the {INSIGHTS_AGENT_TAGS} are present.
   2. click the Logs tab to view the logs the host is ingesting. Logs are only created if actions occur on the software event broker.

Upgrading the Insights Agent

Solace notifies you when new versions of the Insights Agent become available. The notification provides tags required to pull the Insights Agent image from the Solace Container Registry. You should upgrade the Insights Agent when new versions become available.

To upgrade the Insights Agent for Docker and Podman, follow these steps:

1. Pull the new Insights Agent image using the following command, where {INSIGHTS_AGENT_IMAGE_TAG} is the latest (or a specific) version of the Insights Agent.

   docker pull --config /path/to/config/json pull gcr.io/gcp-maas-prod/solace-insights-agent:{INSIGHTS_AGENT_IMAGE_TAG}

2. Stop and remove the Insights Agent container for the previous version.

   docker stop {YOUR_AGENT_CONTAINER_NAME}
   docker rm {YOUR_AGENT_CONTAINER_NAME}

3. Install the new Insights Agent version using the same configuration data and docker run command that you used for the previous installation. You must use the new image tag reference if you are not using latest when creating the new container.

Uninstalling the Insights Agent

You can uninstall the Insights Agent by stopping and removing the containers you created during the Insights Agent installation process. You can uninstall the containers one at a time, or as a group. You should also remove any volumes you created to persist the Insights Agent tailing state, and remove registry credentials.

Uninstalling the Insights Agent does not end your Insights subscription. To end your Insights subscription, contact Solace.

Uninstalling a Single Insights Agent Container

Run the following commands to uninstall a single Insights Agent container:

1. Stop the Insights Agent container:

docker stop {INSIGHTS_AGENT_CONTAINER_NAME}

2. Remove the Insights Agent container:

docker rm {INSIGHTS_AGENT_CONTAINER_NAME}

Uninstalling Multiple Insights Agent Containers

Run the following commands to uninstall multiple Insights Agent containers:

1. Stop all Insights Agent containers:

   docker stop {INSIGHTS_AGENT_CONTAINER_NAME_1} {INSIGHTS_AGENT_CONTAINER_NAME_2} {INSIGHTS_AGENT_CONTAINER_NAME_3}

2. Remove all Insights Agent containers:

   docker rm {INSIGHTS_AGENT_CONTAINER_NAME_1} {INSIGHTS_AGENT_CONTAINER_NAME_2} {INSIGHTS_AGENT_CONTAINER_NAME_3}

Removing Persistent State Volumes

Run the following command for each node to uninstall any volumes you created for Insights Agent state persistence:

docker volume rm <your_agent_container_name>-state