Solace Insights for Self-Managed Container-Based Software Event Broker

You can benefit from the observability features of Solace Insights for your self-managed Solace Software Event Brokers by installing an Insights Agent in your container environment. You can deploy the Insights Agent to both Docker and Podman Container Engines.

After installing and configuring the Insights Agent, you can access and use the numerous Insights dashboards, monitors, and metrics provided by Solace in your Datadog sub-organization. For more information, see Solace Insights Advanced Monitoring.

Before deploying Insights for your Container-based software event broker, review the Considerations for Deploying Insights for Container-Based Software Event Brokers and the Insights Agent Resource Requirements.

The process for installing the Insights Agent to your container-based environment includes three steps:

  1. Downloading the Insights Agent

  2. Configuring and Installing the Insights Agent

  3. Validating the Insights Agent Installation

If you have entitlements for Insights forwarding for self-managed event brokers, you must add extra steps to the configuration process. See Configuring Insights Forwarding for Self-Managed Container-Based Software Event Broker.

After installing the Insights Agent, you must periodically update it when new versions become available. Failure to update the Insights Agent may affect its ability to ingest your software event broker metrics and logs and provide them to Datadog. For more information, see Upgrading the Insights Agent.

You can uninstall the Insights Agent if required. For more information, see Uninstalling the Insights Agent.

Considerations for Deploying Insights for Container-Based Software Event Brokers

Be aware of the following considerations when using Insights with a self-managed software event broker in Docker or Podman:

  • Review the Considerations for Using Insights with Self-Managed Event Brokers.

  • You must use the Add Insights Monitoring form in the Cloud Console to acquire the image pull secret and variable value referenced in the CLI commands in the procedures in this document.

  • Solace recommends securely saving the commands used in the procedures below on a per-installation basis. You will require them in the future when upgrading the Insights Agent.

  • The CLI commands provided throughout this document use 'docker', however, they are identical for both Docker and Podman, unless otherwise noted in the procedures. If installing the Insights Agent to Podman, replace 'docker' in our CLI examples with 'podman'.

  • Your Container Engine must meet these minimum version requirements:

    • Docker version 26.0.0 or greater

    • Podman version 5.4.0 or greater

  • If you have entitlements for Insights forwarding for self-managed event brokers, you must review the additional considerations when deploying the Insights Agent with Insights forwarding.

Insights Agent Resource Requirements

Solace recommends providing each Insights Agent with the following minimum resources when deploying in Docker or Podman:

Resource Type Minimum Requirement
CPU 0.5 CPU cores
Memory 512 MiB

If you are configuring Insights forwarding, the Insights Agent requires additional resources based on your event broker's size class. For more information, see Resource Requirements.

Downloading the Insights Agent

You have two choices for downloading the Insights Agent:

Pulling the Insights Agent Image from the Solace Container Registry

You can configure your deployment to pull the image directly from the Solace Container Registry. You must use the registry secret provided by the Cloud Console (config.json) after completing the Add Insights Monitoring form to pull the image from the Solace Container Registry.

Save the config.json file to a secure location on your Docker host, for example /path/to/config/json.

Use the above path where you saved the config.json file with the --config flag in your docker commands that require pulling the image.

Alternatively, if you don't want to pull images using the docker --config command, you can use the credentials in the config.json file to manually perform a docker login to the Solace Container Registry. Doing so merges the authentication credentials with your existing Docker configuration.

Downloading the Insights Agent from Products

You can download the Insights Agent image tar files from products.solace.com. If you require assistance, contact Solace.

Configuring and Installing the Insights Agent

  • You will need to reuse the configuration files and docker run commands used in the following sections in the future when you upgrade the Insights Agent. Solace recommends securely saving these configuration files and commands on a per-installation basis, in a way that identifies the installation by container name so you can reuse them.

  • If you have entitlements for Insights forwarding, you must perform additional configuration steps when deploying the Insights Agent. For more information, see Configuring Insights Forwarding for Self-Managed Container-Based Software Event Broker.

After downloading and saving the Insights Agent registry secret, you can install the Insights Agent container using the CLI commands below. We provide two examples:

Running either of the commands produces the following results:

  • Runs the Insights Agent in a container named {INSIGHTS_AGENT_CONTAINER_NAME}. The container runs:

    • in detached mode

    • with the default root user ID 0

  • Injects the environment variables required to configure the Insights Agent into its container.

  • Shares the software event broker's /var/lib/solace/jail/logs path with the Insights Agent under /jail/logs.

  • Provides the container with recommended resources, as outlined in Insights Agent Resource Requirements.

  • Pulls the latest version of the Insights Agent image from the Solace Container Registry.

You configure the Insights Agent using the environment variables generated by the Cloud Console after you complete the Add Insights Monitoring form. You can modify the rest of the commands to suit your requirements. We provide guidance for adding the environment variables and updating the commands below. For more information, see Configuring the Installation Command.

You must run the command to install an Insights Agent on every node (messaging and monitoring) in a high-availability event broker, using the environment variables specific to each node.

Using an Environment File to Run the Insights Agent

To run the Insights Agent using an environment file, you must use the event broker properties generated by the Cloud Console after completing the Add Insights Monitoring form to create an environment file. Then, run the following command: 

docker --config /path/to/config/json run \
  --name {INSIGHTS_AGENT_CONTAINER_NAME}     # Descriptive container name \
  -d                                         # Run in detached mode \
  --restart=always                           # Optional: Restart policy \
  --network {SOLACE-NET}                     # Connect to the same network as the broker \
  --env-file {./PRIMARY-NODE.ENV}            # Load environment variables from file \
  # Mount broker logs (read-only) - adjust 'src' and 'volume-subpath' as needed
  --mount type=volume,src={BROKER-STORAGE},target=/jail/logs,volume-subpath=jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended
  --mount type=volume,src={MY-SERVICE-PRIMARY-INSIGHTS-AGENT-STATE},target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" --memory="512m" \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Using Line-by-Line Variable Specification to Run the Insights Agent

To run the Insights Agent using a line-by-line variable specification, you must replace the environment variables (for example, {INSIGHTS_AGENT_TAGS}) in the code snippet below with the environment variables provided by the Cloud Console as KEY=VALUE pairs after completing the Add Insights Monitoring form as -e flags in the command: 

docker --config /path/to/config/json run \
  --name {INSIGHTS_AGENT_CONTAINER_NAME}   # Descriptive container name \
  # Environment variables from Cloud Console (Step 2) \
  -e INSIGHTS_AGENT_API_KEY="{INSIGHTS_AGENT_API_KEY}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_SITE="{INSIGHTS_AGENT_SITE}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_TAGS="{INSIGHTS_AGENT_TAGS}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME="{INSIGHTS_AGENT_BROKER_HOSTNAME}"    # Or IP/Container Name \
  -e INSIGHTS_AGENT_NODE_ROLE="{INSIGHTS_AGENT_NODE_ROLE}" \
  -e INSIGHTS_AGENT_SEMP_USERNAME="{INSIGHTS_AGENT_SEMP_USERNAME}" \
  -e INSIGHTS_AGENT_SEMP_PASSWORD="{INSIGHTS_AGENT_SEMP_PASSWORD}" \
  -e INSIGHTS_AGENT_SEMP_PROTOCOL="{INSIGHTS_AGENT_SEMP_PROTOCOL}" \
  -e INSIGHTS_AGENT_SEMP_PORT="{INSIGHTS_AGENT_PORT}" \
  -e INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION="{INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION}" \
  -e INSIGHTS_AGENT_HEALTH_CHECK_PORT="{INSIGHTS_AGENT_HEALTH_CHECK_PORT}" \
  # Add {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} if needed \
  # Mount broker logs (read-only) \
  --mount type=volume,src={BROKER-STORAGE},target=/jail/logs,volume-subpath=jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended \
  --mount type=volume,src={AGENT-CONTAINER-NAME}-state,target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" --memory="512m" \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Configuring the Installation Command

You can find guidance on configuring the environment variables and other portions of the commands above to meet the specific requirements of your environment in the sections below:

Naming the Insights Agent Container

You must name the Insights Agent container. We recommend using the INSIGHTS_AGENT_BROKER_HOSTNAME variable generated by the Cloud Console after completing the Add Insights Monitoring form, with -insights-agent appended for clarity:

{INSIGHTS_AGENT_CONTAINER_NAME}=INSIGHTS_AGENT_BROKER_HOSTNAME

You can alternatively use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE environment variable, outlined in the Configuring Environment Variables table, to name the Insights Agent container. Using the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE may provide better identifiers for your Insights Agents in Insights.

Networking Configuration Requirements

Your Docker or Podman network must meet the following requirements:

  • If the {INSIGHTS_AGENT_BROKER_HOSTNAME} environment variable is set to your software event broker's container name, then you should run the Insights Agent container on the same Docker or Podman network to allow host resolution with container names. Solace recommends this configuration as a best practice.

  • The Docker or Podman network must allow the Insights Agent to query the software event brokers SEMP endpoints.

Determining the User ID for the Insights Agent

You must run the Insights Agent with a user that has at least read access to the shared log files at /jail/logs. Solace recommends running the Insights Agent with the same user ID (or group ID) used to create the software event broker.

You can omit the -u argument to run the Insights Agent container with UID 0 (root). You must run the container with a UID 0 (root) if you want to persist the Insights Agent log tailing state in a Docker container.

The provided example uses user ID 1001. The User ID can be arbitrary and does not need to correspond to any existing or real user accounts. If the broker's log directory has group-based permissions, you can use a supplemental group ID, which likewise does not have to correspond to any existing or real group IDs. You can provide this supplemental GID in your Docker run command. For example, the following equivalent user run flags use a UID of 1001 and a supplemental GID of 1001:

--user 1001 --group-add 1001

Configuring Environment Variables

You must configure the environment variables using the environment variables generated by the Cloud Console after you complete the Add Insights Monitoring form.

If you are Using an Environment File to Run the Insights Agent you must create an environment file using these values.

The following table lists the environment variables, specifies if the variable is required, how the value is provided (by the Cloud Console or by you, the customer), the purpose of variable, and any additional important information.

Environment Key Variable Required Function Notes, Limitations, and Examples
INSIGHTS_AGENT_API_KEY Yes

Defines the credentials used to authenticate when connecting to the central monitoring service (Datadog) to submit event broker metrics and logs.

Provided by the Cloud Console.

 N/A
INSIGHTS_AGENT_SITE Yes

Defines the central monitoring service (Datadog) region endpoint where Insights ingests event broker metrics and logs.

Provided by the Cloud Console.

 N/A
INSIGHTS_AGENT_TAGS Yes

Metadata attached to your event broker's metrics and logs. These metadata tags uniquely identify the broker node within Insights and associate it with your organization, environment, and service.

Provided by the Cloud Console.

N/A
INSIGHTS_AGENT_BROKER_HOSTNAME Yes

The hostname or IP address that the Insights Agent uses to connect to the event broker node's SEMP and Health Check APIs.

Provided by the Cloud Console.

This variable's value also represents the name for the event broker in your Insights Datadog sub-organization.

For container-based software event broker deployment types, you can optionally use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable to use a custom name in Datadog.

If you do not configure the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable below, this variable must:

  • have a maximum length of 255 characters.

  • be RFC 1123 compliant.

If you do configure the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable below, this variable must be a valid DNS hostname, IP address, or container name.

Examples:

  • my-event-broker-service-primary

  • 10.0.0.1

  • <broker-container-name>

You must not use the following values unless you configure the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable:

  • localhost

  • localhost.localdomain

  • localhost6.localdomain6

  • ip6-localhost
INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE

No

Specified the custom name for the host broker node in Datadog. The value provided for this variable overrides the value in INSIGHTS_AGENT_BROKER_HOSTNAME.

Provided by you, the customer.

While this variable is not mandatory in containerized environments, it is useful for providing identifiers for your event brokers in Insights.

Custom names you provide for the host broker node must meet the following requirements: 

  • Have a maximum length of 255 characters.

  • Must be RFC 1123 compliant.

Examples:

  • my-event-broker-service-primary

  • k8s-node-name-broker-backup

INSIGHTS_AGENT_SEMP_PROTOCOL Yes

Defines the protocol (HTTP or HTTPS) used by the Insights Agent to make SEMP requests to the event broker.

Provided by the Cloud Console.

Possible values are http or https.
INSIGHTS_AGENT_SEMP_PORT Yes

Defines the SEMP port of the event broker used by the Insights Agent.

Provided by the Cloud Console.

Examples could include 8080 or 1943
INSIGHTS_AGENT_SEMP_USERNAME Yes

Defines the read-only management user name used by the Insights Agent when making SEMP requests.

Provided by the Cloud Console.

Software event broker examples: insights

INSIGHTS_AGENT_SEMP_PASSWORD Yes

Defines the password for the read-only management user name defined in the INSIGHTS_AGENT_SEMP_USERNAME variable.

Provided by the Cloud Console.

Software event broker example: password

INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION Yes

Defines whether the Insights Agent should skip TLS certificate verification when making SEMP requests over HTTPS.

Provided by the Cloud Console.

You can set this variable to true if you use self-signed certificates, or don't require validation.

The default value is true.

Possible values are true or false.

INSIGHTS_AGENT_HEALTH_CHECK_PORT Yes

Defines the plain-text HTTP health check port of the event broker node used by the Insights Agent.

Provided by the Cloud Console.

 Example: 5550
INSIGHTS_AGENT_NODE_ROLE Yes

Defines the role of the event broker node (primary, backup, or monitoring) that the Insights Agent is monitoring.

Provided by the Cloud Console.

Possible values:

  • primary or 0

  • backup or 1

  • monitoring or 2

Sharing the Log Files With the Insights Agent

You must give the Insights Agent read-only access to your event brokers log directory, which you can do by mounting the event broker's log directory into the Insights Agent's container at /jail/logs. The path to your event broker's log directory depends on your container setup, but is usually located within the event broker's main data volume; common locations include:

  • /var/lib/solace/jail/logs

  • /usr/sw/jail/logs

You must mount the host path or volume subpath that corresponds to this location. You have two options for configuring this requirement:

Sharing Logs From a Named Docker or Podman Volume
If your event broker uses a named volume (for example, broker-storage) mounted at /var/lib/solace or /usr/sw inside the event broker container, use the following command to mount the volume's /jail/logs sub-path to /jail/logs directory of the Insights Agent.

  • is named {STORAGE-GROUP}

  • is mounted as {STORAGE-GROUP}:/var/lib/solace

--mount type=volume,src={STORAGE-GROUP},target=/jail/logs,volume-subpath=jail/logs,readonly
Sharing Logs From a Host Path Mount
If your software event broker mounts to a host directory (for example, /mnt/solace-data) inside the event broker container, use the following command to mount the volume's /jail/logs sub-path to /jail/logs directory of the Insights Agent. 
--mount type=bind,src=/mnt/solace/jail/logs,target=/jail/logs,readonly

Persisting the Insights Agent’s Log Tailing State

  • For Docker containers, you must run the Insights Agent with a UID 0 (root) to persist the log tailing state. For more information, see Determining the User ID for the Insights Agent.

  • After enabling log tailing, you can ignore any read and write error messages that appear in the Insights Agent's startup logs. The warnings are erroneous, and do not represent actual errors in the Insights Agent's ability to store log tailing information.

You can prevent missing software event broker logs caused by container restarts by configuring the Insights Agent to persist its log position markers in the /opt/datadog-agent/run directory.

In the following commands, {INSIGHTS-AGENT-CONTAINER-NAME} is the name of the Insights Agent container. You must repeat these commands for each Insights Agent.

  1. Create a dedicated volume to allow the Insights Agent to persist its log position.

    2. docker volume create {INSIGHTS-AGENT-CONTAINER-NAME}-state

  2. Mount the volume (read-write) to the Insights Agent.

    • For Docker and Podman, when running the Insights Agent as root (UID 0)

      --mount type=volume,src={INSIGHTS-AGENT-CONTAINER-NAME}-state,target=/opt/datadog-agent/run

    • For Podman, when running as a non-root user:

      --mount type=volume,src={INSIGHTS-AGENT-CONTAINER-NAME}-state,target=/opt/datadog-agent/run,chown

If you configured Sharing Logs From a Host Path Mount, you configure a similar approach by giving the Insights Agent read and write access to the Insights Agent's state directory.

Configuring Insights Forwarding for Self-Managed Container-Based Software Event Broker

If you have an entitlement to Insights forwarding for self-managed event brokers, you must configure the Insights Agent to forward your Insights metrics and logs. You can forward metrics and logs to your Datadog Insights account, to a third-party observability platform, such as Dynatrace, New Relic, Splunk, or other generic OTLP supporting platforms, or to both.

Forwarding your metrics and logs requires you to configure the Insights Agent to mount an additional configuration file (otel-config.yaml) and set specific environment variables. Below are the extra steps you must follow when installing the Insights Agent with Insights forwarding, including:

For example installation commands, see:

For an overview of Insights forwarding for self-managed event brokers, including supported platforms and deployment modes, see Forwarding Solace Insights Metrics and Logs for Self-Managed Event Brokers.

After completing installation of the Insights Agent with Insights forwarding enabled, Solace recommends validating that your metrics and logs are being forwarded.

Solace recommends securely storing the otel-config.yaml file used in this installation process, as you need it for upgrading the Insights Agent.

Considerations for Insights Forwarding

Be aware of the following additional considerations when deploying Insights forwarding:

Mounting the OTel Configuration

You must mount the otel-config.yaml file into the Insights Agent container. If using dual forwarding mode, you must also mount the logs.yaml file. For third-party only mode, the logs.yaml file is not required. The method differs between Docker and Podman:

Mounting the OTel Configuration Using Podman Secrets

Podman supports secrets for secure credential management:

  1. Create a podman secret containing the otel-config.yaml file:

    podman secret create {CONTAINER-NAME}-otel-config /path/to/otel-config.yaml

    Replace {CONTAINER-NAME} with your Insights Agent container name.

  2. If using dual forwarding mode, also create a secret for logs.yaml:

    podman secret create {CONTAINER-NAME}-logs-config /path/to/logs.yaml

  3. Mount the secrets in your podman run command using the --secret flag:

    For third-party only mode: 

    --secret "{CONTAINER-NAME}-otel-config,type=mount,target=/etc/otel/config.yaml"

    For dual forwarding mode (add both secrets): 

    --secret "{CONTAINER-NAME}-otel-config,type=mount,target=/etc/otel/config.yaml" \
--secret "{CONTAINER-NAME}-logs-config,type=mount,target=/etc/datadog-agent/conf.d/solace.d/logs.yml"

  4. After creating secrets, securely delete the plaintext configuration files:

    shred -u /path/to/otel-config.yaml
shred -u /path/to/logs.yaml  # if applicable
Mounting the OTel Configuration for Docker Using Bind Mounts

Docker (non-Swarm mode) does not support secrets, so you must use a bind mount:

  1. Save the otel-config.yaml file to a secure location on your Docker host, for example /etc/solace-insights/otel-config.yaml.

  2. If using dual forwarding mode, also save the logs.yaml file to /etc/solace-insights/logs.yaml.

  3. Set appropriate file permissions to protect the credentials in the files:

    chmod 600 /etc/solace-insights/otel-config.yaml
chmod 600 /etc/solace-insights/logs.yaml  # if applicable

  4. Mount the files in your docker run command:

    For third-party only mode: 

    --mount type=bind,src=/etc/solace-insights/otel-config.yaml,target=/etc/otel/config.yaml,readonly

    For dual forwarding mode (add both mounts): 

    --mount type=bind,src=/etc/solace-insights/otel-config.yaml,target=/etc/otel/config.yaml,readonly \
--mount type=bind,src=/etc/solace-insights/logs.yaml,target=/etc/datadog-agent/conf.d/solace.d/logs.yml,readonly

Configuring Environment Variables

The following section describes the environment variables you can set when enabling Insights forwarding. Add the appropriate variables as -e flags in your Podman or Docker run commands:

Required Core Forwarding Variables

The following environment variables are required for all forwarding modes:

INSIGHTS_AGENT_FORWARDING_ENABLED=true
INSIGHTS_AGENT_BROKER_SIZE={dev|1k|10k|100k|200k}
INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLE={true|false}

The INSIGHTS_AGENT_BROKER_SIZE variable specifies your event broker's size class and automatically derives the appropriate GOMEMLIMIT value for memory management. The valid values are:

  • dev—For STD-100-Dev event brokers (derives a GOMEMLIMIT of 409 MiB)

  • 1k—For STD-1K event brokers (derives a GOMEMLIMIT of 819 MiB)

  • 10k—For ENT-10K event brokers (derives a GOMEMLIMIT of 1638 MiB)

  • 100k—For ENT-100K event brokers (derives a GOMEMLIMIT of 3276 MiB)

  • 200k—For ENT-200K event brokers (derives a GOMEMLIMIT of 4505 MiB)

The INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED variable is required for all forwarding modes. Set this variable to control whether the Insights Agent sends data to your Insights Datadog account in addition to your third-party platform:

  • true — The Insights Agent forwards metrics and logs to both Insights (via Datadog) and your third-party observability platform.

  • false — The Insights Agent forwards metrics and logs to your third-party observability platform only. When set to false, the agent automatically infers DD_LOGS_ENABLED=false and DD_ADDITIONAL_ENDPOINTS={}.

Optional Variables

You can add these optional variables for advanced configurations (see Environment Variables for Insights Forwarding for Self Managed Brokers for a complete reference):

Variable Description
INSIGHTS_AGENT_TELEMETRY_ENABLED Set to false to suppress Datadog /intake errors when using third-party forwarding only mode.
INSIGHTS_AGENT_GOMEMLIMIT Optional override for Go memory soft limit. This value is automatically derived from INSIGHTS_AGENT_BROKER_SIZE. Only use this variable if you need to manually override the auto-calculated value for advanced configurations.
HTTP_PROXY HTTP proxy URL (for example, http://proxy.example.com:8080)
HTTPS_PROXY HTTPS proxy URL (for example, https://proxy.example.com:8443)
NO_PROXY Hosts to bypass proxy (for example, localhost,127.0.0.1)
Dual Forwarding Mode Variables

If using dual forwarding mode (forwarding to both your Insights Datadog account and third-party platform), add these additional variables:

INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED=true
INSIGHTS_AGENT_ADDITIONAL_ENDPOINTS='{"https://app.datadoghq.com":["your-api-key"]}'
INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS='[{"api_key":"your-api-key","host":"agent-http-intake.logs.datadoghq.com","use_compression":true,"compression_level":2}]'

For a complete reference of environment variables, see Environment Variables for Insights Forwarding for Self Managed Brokers.

Example of a Docker Run Command with Insights Forwarding Enabled

The --memory value shown in the following example (768m) is for a STD-100-Dev event broker. See the Resource Requirements table and adjust the number based on your event broker size class.

The following example runs the Insights Agent with Insights forwarding in "third-party only" mode on an software event broker deployed to a Docker container using bind mounts. You can use this example instead of the line-by-line variable specification for installing the Insights Agent.

Step 1: Create the Environment File

Create an environment file named insights-agent.env containing all the required variables. Replace the placeholder values with your actual configuration from the Cloud Console:

# Core forwarding variables
INSIGHTS_AGENT_THIRD_PARTY_FORWARDING_ENABLED=true
INSIGHTS_AGENT_BROKER_SIZE=dev

# Insights forwarding toggle (set to false for third-party only mode)
INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED=false

# Standard variables from Cloud Console
INSIGHTS_AGENT_TAGS={INSIGHTS_AGENT_TAGS}
INSIGHTS_AGENT_BROKER_HOSTNAME={BROKER_CONTAINER_NAME}
INSIGHTS_AGENT_NODE_ROLE=primary
INSIGHTS_AGENT_SEMP_PORT=8080
INSIGHTS_AGENT_SEMP_USERNAME={INSIGHTS_AGENT_SEMP_USERNAME}
INSIGHTS_AGENT_SEMP_PASSWORD={INSIGHTS_AGENT_SEMP_PASSWORD}
INSIGHTS_AGENT_SEMP_PROTOCOL=http
INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION=true
INSIGHTS_AGENT_HEALTH_CHECK_PORT=5550

# For dual forwarding mode, set INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED=true
# and add these variables:
# INSIGHTS_AGENT_ADDITIONAL_ENDPOINTS={"https://app.datadoghq.com":["{DD_API_KEY}"]}
# INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS=[{"api_key":"{DD_API_KEY}","host":"agent-http-intake.logs.datadoghq.com","use_compression":true,"compression_level":2}]

INSIGHTS_AGENT_BROKER_HOSTNAME must be the broker container's DNS name on the shared network (for example, solace-broker), not localhost.

Step 2: Run the Container

Run the Insights Agent container using the environment file:

docker run -d --restart=always \
--name <insights_agent_podname> \
--network <network_name> \
--env-file <insights_agent_env_file> \
--mount type=volume,src=<insights_agent_state_volume_name>,target=/opt/datadog-agent/run \
--mount type=volume,src=<broker_volume_name>,target=/jail/logs,volume-subpath=jail/logs,readonly \
--mount type=bind,source=<otel_config_file>,target=/etc/datadog-agent/otel-config.yaml,readonly \
--cpus="0.5" --memory="768m" \
gcr.io/gcp-maas-prod/solace-insights-agent:latest

For dual forwarding mode, add the logs configuration mount:

--mount type=bind,source=<logs_config_file>,target=/etc/datadog-agent/conf.d/solace.d/logs.yml,readonly \
Step 3: Remove the Environment File

The environment file contains sensitive credentials. Remove it after the container starts.

Example of Podman Run Command with Insights Forwarding Enabled

The --memory value shown in the following example (768m) is for a STD-100-Dev event broker. See the Resource Requirements table and adjust the number based on your event broker size class.

The following example runs the Insights Agent with Insights forwarding in "third-party only" mode on an software event broker deployed to a Podman container. You can use this example instead of the line-by-line variable specification for installing the Insights Agent without forwarding.

Step 1: Create the Environment File

Create an environment file (for example, insights-agent.env) containing all the required variables. Replace the placeholder values with your actual configuration from the Cloud Console:

# Core forwarding variables
INSIGHTS_AGENT_THIRD_PARTY_FORWARDING_ENABLED=true
INSIGHTS_AGENT_BROKER_SIZE=dev

# Insights forwarding toggle (set to false for third-party only mode)
INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED=false

# Standard variables from Cloud Console
INSIGHTS_AGENT_TAGS={INSIGHTS_AGENT_TAGS}
INSIGHTS_AGENT_BROKER_HOSTNAME={BROKER_CONTAINER_NAME}
INSIGHTS_AGENT_NODE_ROLE=primary
INSIGHTS_AGENT_SEMP_PORT=8080
INSIGHTS_AGENT_SEMP_USERNAME={INSIGHTS_AGENT_SEMP_USERNAME}
INSIGHTS_AGENT_SEMP_PASSWORD={INSIGHTS_AGENT_SEMP_PASSWORD}
INSIGHTS_AGENT_SEMP_PROTOCOL=http
INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION=true
INSIGHTS_AGENT_HEALTH_CHECK_PORT=5550

# For dual forwarding mode, set INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED=true
# and add these variables:
# INSIGHTS_AGENT_ADDITIONAL_ENDPOINTS={"https://app.datadoghq.com":["{DD_API_KEY}"]}
# INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS=[{"api_key":"{DD_API_KEY}","host":"agent-http-intake.logs.datadoghq.com","use_compression":true,"compression_level":2}]

INSIGHTS_AGENT_BROKER_HOSTNAME must be the broker container's DNS name on the shared network (for example, solace-broker), not localhost.

Step 2: Run the Container

Run the Insights Agent container using the environment file:

podman run -d --restart=always \
--name <insights_agent_podname> \
--network <network_name> \
--env-file <insights_agent_env_file> \
--mount type=volume,src=<insights_agent_state_volume_name>,target=/opt/datadog-agent/run \
--mount type=volume,src=<broker_volume_name>,target=/jail/logs,volume-subpath=jail/logs,readonly \
--secret <insights_agent_podname>-otel-config,type=mount,target=/etc/datadog-agent/otel-config.yaml,mode=0644 \
--cpus="0.5" --memory="768m" \
gcr.io/gcp-maas-prod/solace-insights-agent:latest

For dual forwarding mode, add the logs configuration secret:

--secret <insights_agent_podname>-logs-config,type=mount,target=/etc/datadog-agent/conf.d/solace.d/logs.yml,mode=0644 \
Step 3: Remove the Environment File

The environment file contains sensitive credentials. Remove it after the container starts.

Validating the Insights Agent Installation

After configuring the Insights Agent, you can validate the Insights Agent is working by logging into your Insights Datadog account and viewing the Insights Agent host in the Datadog infrastructure list.

If you have configured Insights forwarding to a third party observability platform, you should also validate that your Insights metrics and logs are reaching that platform.

To view the Insights Agent host in the Datadog infrastructure list, do the following:

  1. Log in to the Solace Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the Solace Cloud Console.
  2. Select Insights on the navigation bar.
  3. Click View in Datadog from any of the available tiles where it appears.
  4. If prompted to do so, log in to Datadog.
  5. From the menu on the left, select Infrastructure and then click Hosts.
  6. In the Search by field, enter one of the following values to filter the list to show only your Insights Agent host:
    • {INSIGHTS_AGENT_BROKER_HOSTNAME} where {INSIGHTS_AGENT_BROKER_HOSTNAME} is the name of the Insights Agent.
    • {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} where {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} is the name of the Insights Agent if you set this environment variable.
  7. Select the host to access its information. From this screen you can:
    1. click the Host Info tab to confirm the {INSIGHTS_AGENT_TAGS} are present.
    2. click the Logs tab to view the logs the host is ingesting. Logs are only created if actions occur on the software event broker.

Validating Insights Forwarding

After deploying with Insights forwarding enabled, verify that metrics and logs are reaching your third-party platform:

  1. Check the Insights Agent container logs to verify the OTel collector started successfully:

    docker logs {INSIGHTS_AGENT_CONTAINER_NAME}
# or for Podman:
podman logs {INSIGHTS_AGENT_CONTAINER_NAME}

    Look for log entries indicating the solace-otel-agent process has started. If you see the message "third party forwarding is not configured, falling back silently to previous datadog-only telemetry", the OTel configuration is invalid or missing.

  2. Verify the Insights Agent health check is responding:

    curl http://localhost:5550/health

  3. Verify environment variables are set correctly using the process environment:

    strings /proc/$(docker exec {INSIGHTS_AGENT_CONTAINER_NAME} pgrep -f datadog-agent)/environ | grep INSIGHTS
# or for Podman:
strings /proc/$(podman exec {INSIGHTS_AGENT_CONTAINER_NAME} pgrep -f datadog-agent)/environ | grep INSIGHTS

    Confirm that INSIGHTS_AGENT_FORWARDING_ENABLED=true and INSIGHTS_AGENT_OTEL_CONFIG_FILE is set to the correct path.

  4. Log into your third-party observability platform and verify metrics are appearing:

    • Look for metrics with the prefix solace. or pubsubplus.

    • Check for the datadog.agent.running heartbeat metric (emitted approximately every 60 seconds)

    • Verify your event broker hostname appears in the metric metadata

If your Insights metrics and logs are not reaching your third-party observability platform, try the troubleshooting steps below.

Troubleshooting Insights Forwarding

You can try the following commands if your Insights metrics or logs do not appear in your third-party observability platform:

  • Verify the otel-config.yaml file is accessible in the container: 

    docker exec {INSIGHTS_AGENT_CONTAINER_NAME} cat /etc/otel/config.yaml
# or for Podman:
podman exec {INSIGHTS_AGENT_CONTAINER_NAME} cat /etc/otel/config.yaml

  • Check the Insights Agent logs for OTel collector errors: 

    docker logs {INSIGHTS_AGENT_CONTAINER_NAME} | grep -i error
# or for Podman:
podman logs {INSIGHTS_AGENT_CONTAINER_NAME} | grep -i error

  • Verify environment variables are set correctly: 

    docker exec {INSIGHTS_AGENT_CONTAINER_NAME} env | grep INSIGHTS_AGENT
# or for Podman:
podman exec {INSIGHTS_AGENT_CONTAINER_NAME} env | grep INSIGHTS_AGENT

  • Verify network connectivity to your third-party platform: 

    docker exec {INSIGHTS_AGENT_CONTAINER_NAME} curl -I https://your-observability-platform.com
# or for Podman:
podman exec {INSIGHTS_AGENT_CONTAINER_NAME} curl -I https://your-observability-platform.com

  • Ensure the agent image version supports third-party forwarding: Verify you're using image version 5d35eea or newer. Do not use :latest as it may reference an older version.

  • For Docker bind mounts, verify file permissions: Ensure the otel-config.yaml file is readable by the container user (typically root or the UID specified with --user).

  • For Podman secrets, verify the secret exists: 

    podman secret ls | grep otel-config

  • Set up ongoing health monitoring: To proactively detect Insights Agent health issues, configure monitoring for the datadog.agent.running heartbeat metric in your observability platform. For more information, see Monitoring Insights Agent Health for Self-Managed Event Brokers.

Upgrading the Insights Agent

  • Solace recommends creating a downtime in your Datadog account before upgrading the Insights Agent, to prevent no data alerts while the Insights Agent is not running. You should scope this downtime to service_name for All Monitors. For more information, see Scheduling Downtime.

  • When upgrading the Insights Agent, you must use the same otel-config.yaml file and environment variables from your original installation. Do not regenerate the configuration unless you are also changing your third-party platform settings.

Solace notifies you when new versions of the Insights Agent become available. The notification provides tags required to pull the Insights Agent image from the Solace Container Registry. You should upgrade the Insights Agent when new versions become available.

To upgrade the Insights Agent for Docker and Podman, follow these steps:

  1. Pull the new Insights Agent image using the following command, where {INSIGHTS_AGENT_IMAGE_TAG} is the latest (or a specific) version of the Insights Agent.

    docker --config /path/to/config/json pull gcr.io/gcp-maas-prod/solace-insights-agent:{INSIGHTS_AGENT_IMAGE_TAG}

  2. Stop and remove the Insights Agent container for the previous version. 

    docker stop {YOUR_AGENT_CONTAINER_NAME}
docker rm {YOUR_AGENT_CONTAINER_NAME}

  3. Install the new Insights Agent version using the same configuration data and docker run command that you used for the previous installation. You must use the new image tag reference if you are not using latest when creating the new container.

Uninstalling the Insights Agent

Solace recommends creating a downtime in your Datadog account before uninstalling the Insights Agent, to prevent erroneous alerts while the Insights Agent is not running. This downtime should be scoped to service_name for All Monitors. For more information, see Scheduling Downtime.

You can uninstall the Insights Agent by stopping and removing the containers you created during the Insights Agent installation process. You can uninstall the containers one at a time, or as a group. You should also remove any volumes you created to persist the Insights Agent tailing state, and remove registry credentials.

Uninstalling the Insights Agent does not end your Insights subscription. To end your Insights subscription, contact Solace.

Uninstalling a Single Insights Agent Container

Run the following commands to uninstall a single Insights Agent container:

  1. Stop the Insights Agent container:

    2. docker stop {INSIGHTS_AGENT_CONTAINER_NAME}

  2. Remove the Insights Agent container:

    3. docker rm {INSIGHTS_AGENT_CONTAINER_NAME}

Uninstalling Multiple Insights Agent Containers

Run the following commands to uninstall multiple Insights Agent containers:

  1. Stop all Insights Agent containers:

    docker stop {INSIGHTS_AGENT_CONTAINER_NAME_1} {INSIGHTS_AGENT_CONTAINER_NAME_2} {INSIGHTS_AGENT_CONTAINER_NAME_3}

  2. Remove all Insights Agent containers:

    docker rm {INSIGHTS_AGENT_CONTAINER_NAME_1} {INSIGHTS_AGENT_CONTAINER_NAME_2} {INSIGHTS_AGENT_CONTAINER_NAME_3}

Removing Persistent State Volumes

Run the following command for each node to uninstall any volumes you created for Insights Agent state persistence:

docker volume rm <your_agent_container_name>-state