Solace Insights for Self-Managed Appliance Event Brokers

Using Insights for self-managed appliance event broker's is available as a Controlled Availability (CA) feature in Solace Cloud. Contact Solace to evaluate if this option meets your use case and to get more information.

You can benefit from the observability features of Solace Insights for your self-managed Solace Appliance Event Broker by installing an Insights Agent on your appliance event broker. This feature is offered as controlled availability.

After installing and configuring the Insights Agent, you can access and use the numerous Insights dashboards, monitors, and metrics provided by Solace in your Datadog sub-organization. For more information, see Solace Insights Advanced Monitoring.

Before deploying Insights for your appliance event broker, review the Considerations for Deploying Insights for Appliance Event Brokers.

The process for installing the Insights Agent includes three steps:

  1. Preparing the Appliance Event Broker for Insights Agent Installation

  2. Configuring and Installing the Insights Agent

  3. Validating the Insights Agent Installation

If you are installing the Insights Agent on appliance event brokers in a high availability (HA) configuration, you must repeat this process for each appliance event broker in the HA pair using the environment variables specific to each node.

If you have entitlements for Insights forwarding for self-managed event brokers, you must add extra steps to the configuration process. See Configuring Insights Forwarding for Self-Managed Appliance Event Brokers.

After installing the Insights Agent, you must periodically update it when new versions become available. Failure to update the Insights Agent may affect its ability to ingest your appliance event broker's metrics and logs and provide them to Datadog. For more information, see Upgrading the Insights Agent.

You can uninstall the Insights Agent if required. For more information, see Uninstalling the Insights Agent.

Considerations for Deploying Insights for Appliance Event Brokers

Be aware of the following considerations when using Insights with a self-managed Appliance Event Broker:

  • Review the Considerations for Using Insights with Self-Managed Event Brokers.

  • Your appliance event broker must be running version soltr_10.25.3.2742 or later with Podman version 5.4.0 or later installed. For more information, see Changing Appliance Event Broker Software.

  • The Insights Agent runs containerized service using Podman on the appliance event broker with the insights user account.

  • You must configure a dedicated Solace Event Broker CLI user with global read permissions that uses basic authentication on the appliance event broker. For more information, see Configuring a SEMP User for the Insights Agent.

  • You must configure your appliance event broker before installing the Insights Agent. Preparing your appliance event broker includes stopping the Solace Geneos Agent, if running. For more information, see Preparing the Appliance Event Broker for Insights Agent Installation

  • The Insights Agent connects to the appliance event broker using HTTP on localhost (127.0.0.1) for SEMP requests.

  • Your appliance event broker must have DNS configured correctly to reach gcr.io and your Datadog site (e.g., datadoghq.com or datadoghq.eu).

  • You must use the Add Insights Monitoring form in the Cloud Console to acquire the image pull secret and API key referenced in the CLI commands in the procedures in this document. For more information, see Insights for Self-Managed Event Brokers.

  • Solace recommends securely saving the commands used in the procedures below on a per-installation basis. You will require them in the future when upgrading the Insights Agent.

  • If you have entitlements for Insights forwarding for self-managed event brokers, you must review the additional considerations when deploying the Insights Agent with Insights forwarding.

Insights Agent Metric Collection Limitations on Appliance event brokers

While the Insights Agent provides the ability to collect most logs and metrics from your appliance event broker, it cannot collect the following metrics for the default Message VPN:

  • derived-metrics.client.connection.authentication.client-cert-auth.enabled

  • derived-metrics.client.connection.authentication.internal.enabled

  • derived-metrics.client.connection.authentication.ldap.enabled

  • derived-metrics.client.connection.authentication.kerberos-auth.enabled

  • derived-metrics.client.connection.authentication.oauth.enabled

Insights Agent Resource Requirements

The Insights Agent requires the following resources from your appliance event broker:

Resource Type Minimum Requirement
CPU 0.5 CPU cores
Memory 512 MiB
CPU Set CPU core 4 (isolated)

If you are configuring Insights forwarding for your appliance event broker, you must provide additional resource requirements based on your event broker's size class.

Preparing the Appliance Event Broker for Insights Agent Installation

If you have entitlements for Insights forwarding, you must perform additional configuration steps when deploying the Insights Agent. For more information, see Configuring Insights Forwarding for Self-Managed Appliance Event Brokers.

Before installing the Insights Agent on your appliance event broker, you must perform several steps to ensure you appliance event broker is ready. These steps include:

  1. Configuring a SEMP User for the Insights Agent

  2. Accessing the appliance event broker Linux shell and:

    1. Stopping the Solace Geneos Agent

    2. Switching to the Insights User

  3. Solace Insights for Self-Managed Appliance Event Brokers

  4. Solace Insights for Self-Managed Appliance Event Brokers

If you are installing the Insights Agent on appliance event brokers in a high availability (HA) configuration, you must repeat this process for each appliance event broker in the HA pair.

Configuring a SEMP User for the Insights Agent

You must create a SEMP user for the Insights Agent so it can send SEMP requests. The user should be provided with the minimal required permissions to monitor all necessary metrics on the appliance event broker, in this case, global read only permissions. For more information see CLI User Access Levels.

Log into the Solace Event Broker CLI on your appliance event broker and use the following CLI command to create the SEMP user on your appliance event broker:

solace# enable
solace# configure
solace(configure)# create username insights-agent password <password> cli global-access-level read-only
solace(configure/username)# message-vpn default-access-level read-only
solace(configure/username)# end

Accessing the Appliance Event Broker Linux Shell

After creating the SEMP user on the appliance event broker, you must access the Linux shell on the appliance event broker to continue configuring your appliance event broker for installation of the Insights Agent.

From the Linux shell, you must:

  1. stop the Solace Geneos Agent if it is running

  2. switch to the Insights user

  3. configure the Insights Agent download

For more information, see Accessing the Linux Shell Through Solace Event Broker CLI.

Stopping the Solace Geneos Agent

You cannot run Solace Geneos Agent and the Insights Agent at the same time. If you are running Solace Geneos Agent on your appliance event broker, you must stop it to prevent conflicts with the Insights Agent. Run the following Linux shell commands to stop the Solace Geneos Agent: 

systemctl status solgeneos
systemctl stop solgeneos

Switching to the Insights User

You must switch to the insights user to continue to install the Insights Agent. Run the following commands to switch and verify that you are the Insights user:

su - insights
id  # Verify you are now the insights user

Configuring the Insights Agent Download

You have two choices for downloading the Insights Agent:

Pulling the Insights Agent Image from the Solace Container Registry

You can configure your deployment to pull the image directly from the Solace Container Registry. You must use the registry secret provided by the Cloud Console (config.json) after completing the Add Insights Monitoring form to pull the image from the Solace Container Registry.

  1. Log into the appliance event broker Linux shell if you haven't done so and switch to the insights Linux shell user. For more information, see Accessing the Linux Shell Through Solace Event Broker CLI, and Switching to the Insights User.

  2. Place the config.json file to the /usr/sw/insights/.docker/config.json path on the appliance event broker.

    scp <path_to_config_json>/config.json root@lab-128-82:/usr/sw/insights/.docker/config.json

  3. You must set ownership of the config.json file to the insights user on the appliance event broker. Use the following command to set ownership of the config.json file:

    chown insights:insights /usr/sw/insights/.docker/config.json

Use the directory where you saved the config.json file in step one with the --config flag in your podman commands that require pulling the image for installing the Insights Agent.

Downloading the Insights Agent from Products

If you don't want to provide access for your appliance event broker to the Solace Container Registry, you can download the Insights Agent image tar files from products.solace.com. If you require assistance, contact Solace.

Configuring and Installing the Insights Agent

You will need to reuse the configuration files and podman run commands used in the following sections in the future when you upgrade the Insights Agent. Solace recommends securely saving these configuration files and commands on a per-installation basis, in a way that identifies the installation by container name so you can reuse them.

After preparing your appliance event broker for installation of the Insights Agent, you can install the Insights Agent to the Podman container on the appliance event broker. We provide two examples:

Running either of the commands produces the following results:

  • Injects the environment variables required to configure the Insights Agent into its container.

  • Shares the appliance event broker's /usr/sw/jail/logs path with the Insights Agent under /jail/logs.

  • Provides the container with recommended resources, as outlined in Insights Agent Resource Requirements.

  • Pulls the latest version of the Insights Agent image from the Solace Container Registry.

You configure the Insights Agent using the environment variables generated by the Cloud Console after you compete the Add Insights Monitoring form. You can modify the rest of the commands to suit your requirements. We provide guidance for adding the environment variables and updating the commands below. For more information, see Configuring the Installation Command.

If you are installing the Insights Agent on appliance event brokers in a high availability (HA) configuration, you must repeat this process for each appliance event broker in the HA pair, using the environment variables specific to each node.

Using an Environment File to Run the Insights Agent

To run the Insights Agent using an environment file, you must use the event broker properties generated by the Cloud Console after completing the Add Insights Monitoring form to create an environment file. As the insights Linux shell user, run the following command: 

podman --config /path/to/config/json run \
  --name {INSIGHTS_AGENT_CONTAINER_NAME}     # Descriptive container name \
  -d                                         # Run in detached mode \
  --restart=always                           # Optional: Restart policy \
  --network host                             
  --env-file ./primary-node.env              # Load environment variables from file \
  --secret "insights-agent-semp-password,type=env,target={INSIGHTS_AGENT_SEMP_PASSWORD}" \
  --secret "insights-agent-api-key,type=env,target={INSIGHTS_AGENT_API_KEY}" \
  # Mount broker logs (read-only)
  --mount type=bind,source=/usr/sw/jail/logs/,destination=/jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended
  --mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" \
  --memory="512m" \
  --cpuset-cpus=4 \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Using Line-by-Line Variable Specification to Run the Insights Agent

To run the Insights Agent using a line-by-line variable specification, you must replace the environment variables (for example, {INSIGHTS_AGENT_TAGS}) in the code snippet below with the environment variables provided by the Cloud Console as KEY=VALUE pairs after completing the Add Insights Monitoring form as -e flags in the command. As the insights Linux shell user, run the following command: 

podman --config /path/to/config/json run \
  - d \
  --restart=always \
  --name="my-primary-insights-container" # Descriptive container name \
  --network host \
  # Environment variables from Cloud Console \
  -e INSIGHTS_AGENT_SITE="{INSIGHTS_AGENT_SITE}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_TAGS="{INSIGHTS_AGENT_TAGS}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME="127.0.0.1" \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE="my-service-primary"
  -e INSIGHTS_AGENT_NODE_ROLE="primary" \
  -e INSIGHTS_AGENT_SEMP_USERNAME="insights-agent" \  
  -e INSIGHTS_AGENT_SEMP_PROTOCOL="http" \
  -e INSIGHTS_AGENT_SEMP_PORT="80" \
  -e INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION="true" \
  --secret "insights-agent-semp-password,type=env,target="{INSIGHTS_AGENT_SEMP_PASSWORD}" \
  --secret "insights-agent-api-key,type=env,target="{INSIGHTS_AGENT_API_KEY}" \
  # Mount broker logs (read-only) \
  --mount type=bind,source=/usr/sw/jail/logs/,destination=/jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended \
  --mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" \
  --memory="512m" \
  --cpuset-cpus=4 \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Configuring the Installation Command

You can find guidance on configuring the environment variables and other portions of the commands above to meet the specific requirements of your environment in the sections below:

Configuring Secrets

You must have pass installed and initialized on your appliance event broker before completing the installation of the Insights Agent. For more information, see Solace Insights for Self-Managed Appliance Event Brokers. With pass installed and initialized, you must configure the Insights API key and SEMP password as secretes in pass.

Use the following commands to create the secrets 

podman secret create "insights-agent-semp-password"
podman secret create "insights-agent-api-key"

Use the following commands to populate the secrets with the values provided by completing the Add Insights Monitoring form in the Cloud Console.

--secret "insights-agent-semp-password,type=env,target=INSIGHTS_AGENT_SEMP_PASSWORD"
--secret "insights-agent-api-key,type=env,target=INSIGHTS_AGENT_API_KEY"

Sharing the Log Files With the Insights Agent

You must give the Insights Agent read-only access to your event brokers log directory, which you can do by mounting the event broker's log directory into the Insights Agent's container at /jail/logs. The direct path to the broker's log directory depends on the broker container setup on your appliance event broker, common locations inside the broker container include:

  • /var/lib/solace/jail/logs

  • /usr/sw/jail/logs

Use the following command to the mount the appliance's log directory to the /jail/logs directory of the Insights Agent:

--mount type=bind,src={path to broker's log directory},target=/jail/logs,readonly

Collecting the Insights Agent Logs

You must enable log collection for the Insights Agent logs. To enable Insights Agent log collection set the INSIGHTS_AGENT_LOG_FORWARDING_ENABLED environment variable to 'true'.

Naming the Insights Agent Container

The Add Insights Monitoring form provides an IP address for the hostname of the Insights Agent container (127.0.0.1). You must use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable outlined in the Configuring Environment Variables table, to give a custom hostname to the Insights Agent container. The custom hostname appears as the display name for your appliance in Datadog, providing better identifiers for your Insights Agents.

Use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable generated by the Cloud Console after completing the Add Insights Monitoring form, with -insights-agent appended for clarity.

Persisting the Insights Agent’s Log Tailing State

After enabling log tailing, you can ignore any read and write error messages that appear in the Insights Agent's startup logs. The warnings are erroneous, and do not represent actual errors in the Insights Agent's ability to store log tailing information.

You can prevent missing appliance event broker logs caused by container restarts by configuring the Insights Agent to persist its log position markers in the /opt/datadog-agent/run directory.

To persist the Insights Agent's log tailing state, run the following command:

--mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run

Configuring Environment Variables

You must configure the environment variables using the environment variables generated by the Cloud Console after you complete the Add Insights Monitoring form.

If you are Using an Environment File to Run the Insights Agent you must create an environment file using these values.

The following table lists the environment variables, specifies if the variable is required, how the value is provided (by the Cloud Console or by you, the customer), the purpose of variable, and any additional important information.

Environment Key Variable Required Function Notes, Limitations, and Examples
INSIGHTS_AGENT_API_KEY Yes

Defines the credentials used to authenticate when connecting to the central monitoring service (Datadog) to submit event broker metrics and logs.

Provided by the Cloud Console.

 N/A
INSIGHTS_AGENT_SITE Yes

Defines the central monitoring service (Datadog) region endpoint where Insights ingests event broker metrics and logs.

Provided by the Cloud Console.

 N/A
INSIGHTS_AGENT_TAGS Yes

Metadata attached to your event broker's metrics and logs. These metadata tags uniquely identify the broker node within Insights and associate it with your organization, environment, and service.

Provided by the Cloud Console.

N/A
INSIGHTS_AGENT_BROKER_HOSTNAME Yes

The hostname or IP address that the Insights Agent uses to connect to the event broker node's SEMP and Health Check APIs.

For appliance event broker's the value is always 127.0.0.1

This variable's value also represents the name for the event broker in your Insights Datadog sub-organization.

For appliance event broker-based deployments, this value must be 127.0.0.1.

You can use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable to use a custom name in Datadog.
INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE

Yes

Specified the custom name for the host broker node in Datadog. The value provided for this variable overrides the value in INSIGHTS_AGENT_BROKER_HOSTNAME.

Provided by you, the customer.

Custom names you provide for the host broker node must meet the following requirements: 

  • Have a maximum length of 255 characters.

  • Must be RFC 1123 compliant.

Examples:

  • my-event-broker-service-primary

  • k8s-node-name-broker-backup

INSIGHTS_AGENT_SEMP_PROTOCOL Yes

Defines the protocol (HTTP or HTTPS) used by the Insights Agent to make SEMP requests to the event broker.

Provided by the Cloud Console.

Possible values are http or https.
INSIGHTS_AGENT_SEMP_PORT Yes

Defines the SEMP port of the event broker used by the Insights Agent.

Provided by the Cloud Console.

Examples could include 8080 or 1943
INSIGHTS_AGENT_SEMP_USERNAME Yes

Defines the read-only management user name used by the Insights Agent when making SEMP requests.

Provided by the Cloud Console.

Appliance event broker example: sempread

INSIGHTS_AGENT_SEMP_PASSWORD Yes

Defines the password for the read-only management user name defined in the INSIGHTS_AGENT_SEMP_USERNAME variable.

Provided by the Cloud Console.

Appliance event broker example: sempreadpassword

INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION Yes

The Insights Agent must always skip TLS cuertificate validation when making SEMP requests over HTTPS.

The Insights Agent uses 127.0.0.1 as the hostname to send SEMP requests to the event broker. The event broker's certificate is issued for a hostname other than localhost, which will cause a validation mismatch if TLS verification is not disabled.

Provided by the Cloud Console.

The value must be true.
INSIGHTS_AGENT_NODE_ROLE Yes

Defines the role of the event broker node (primary, backup, or monitoring) that the Insights Agent is monitoring.

Provided by the Cloud Console.

Possible values:

  • primary or 0

  • backup or 1

  • monitoring or 2

Configuring Insights Forwarding for Self-Managed Appliance Event Brokers

If you have an entitlement to Insights forwarding for self-managed event brokers, you must configure the Insights Agent to forward your Insights metrics and logs. You can forward metrics and logs to your Datadog Insights account, to a third-party observability platform, such as Dynatrace, New Relic, Splunk, or other generic OTLP supporting platforms, or to both.

Insights forwarding has numerous requirements and configuration options beyond those of standard Insights for self-managed event brokers. For an overview of Insights forwarding for self-managed event brokers, including supported platforms and deployment modes, see Forwarding Solace Insights Metrics and Logs for Self-Managed Event Brokers.

Forwarding your metrics and logs requires you to configure the Insights Agent to mount an additional configuration file (otel-config.yaml) and set specific environment variables. Below are the extra steps you must follow when installing the Insights Agent with Insights forwarding, including:

For an example command, see Example of Podman Run Command with Insights Forwarding Enabled.

After completing installation of the Insights Agent with Insights forwarding enabled, Solace recommends validating that your metrics and logs are being forwarded.

Solace recommends securely storing the otel-config.yaml file used in this installation process, as you need it for upgrading the Insights Agent.

Additional Considerations for Insights Forwarding

Be aware of the following additional considerations when deploying Insights forwarding:

Deploying the OTel Configuration Files

You must deploy the otel-config.yaml file on your appliance event broker where the Insights Agent can access it. If using dual forwarding mode, you must also deploy the logs.yaml file. To deploy the OTel configuration files, perform these steps: 

  1. As the insights user on the appliance event broker, create the Insights configuration directory if it doesn't exist:

    mkdir -p /usr/sw/insights

  2. Copy the otel-config.yaml file from your downloaded configuration package to the appliance event broker:

    scp /path/to/otel-config.yaml insights@{APPLIANCE_IP}:/usr/sw/insights/otel-config.yaml

    Replace {APPLIANCE_IP} with your appliance event broker's IP address or hostname.

  3. If using dual forwarding mode, copy the logs.yaml file:

    scp /path/to/logs.yaml insights@{APPLIANCE_IP}:/usr/sw/insights/logs.yaml

  4. As the insights user, verify the file ownership and permissions:

    ls -l /usr/sw/insights/otel-config.yaml
ls -l /usr/sw/insights/logs.yaml  # if applicable

    The files must be owned by the insights user and have restrictive permissions (e.g., -rw-------).

Creating Podman Secrets for Insights Forwarding

In addition to the standard secrets for the SEMP password and Datadog API key, Insights forwarding uses additional podman secrets for storing configuration data. As the insights user, create the podman secret for the OTel configuration, replacing the {NAME_PREFIX} variable with a descriptive prefix that matches your container name (for example, my-service-primary).

For third-party only mode: 

podman secret create {NAME_PREFIX}-otel-config /usr/sw/insights/otel-config.yaml

For dual forwarding mode (create both secrets): 

podman secret create {NAME_PREFIX}-otel-config /usr/sw/insights/otel-config.yaml
podman secret create {NAME_PREFIX}-logs-config /usr/sw/insights/logs.yaml

After creating the secrets, securely delete the plaintext configuration files:

shred -u /usr/sw/insights/otel-config.yaml
shred -u /usr/sw/insights/logs.yaml  # if applicable

Configuring Environment Variables

The following section describes the environment variables you can set when enabling Insights forwarding. Add the variables as -e flags in your Podman run command.

Core Forwarding Environment Variables

The following environment variables are required for all Insights forwarding modes:

INSIGHTS_AGENT_THIRD_PARTY_FORWARDING_ENABLED=true
INSIGHTS_AGENT_BROKER_SIZE=APPLIANCE

The INSIGHTS_AGENT_BROKER_SIZE variable specifies your event broker's class size and automatically derives the appropriate GOMEMLIMIT value for memory management. For appliance event brokers, set this value to APPLIANCE, which derives a GOMEMLIMIT of 5888 MiB.

Optional Insights Forwarding Toggle

Use the INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED variable to control whether the Insights Agent sends data to your Insights Datadog account, in addition to your third-party platform:

  • true (default)—Dual-write mode: The Insights Agent forwards metrics and logs to both your Insights Datadog account and your third-party observability platform.

  • false—Third-party only mode: The Insights Agent forwards metrics and logs to your third-party observability platform only. When set to false, the agent automatically infers DD_LOGS_ENABLED=false, DD_ADDITIONAL_ENDPOINTS={}, and INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS=[].

Optional Variables

You can add these additional variables for advanced configurations (see Environment Variables for Insights Forwarding for Self Managed Brokers for complete reference):

  • INSIGHTS_AGENT_TELEMETRY_ENABLED - Set to false in air-gapped environments to suppress Datadog agent error logs related to the /intake endpoint

  • INSIGHTS_AGENT_GOMEMLIMIT - Optional override for the Go memory soft limit. This value is automatically derived from INSIGHTS_AGENT_BROKER_SIZE. You should only use this variable if you need to manually override the auto-calculated value for an advanced configuration.

  • HTTP_PROXY, HTTPS_PROXY, NO_PROXY - Proxy configuration

  • DD_AC_EXCLUDE=name:docker, DD_URL=http://localhost - Air-gapped deployment connection attempt suppression

Dual Forwarding Mode Variables

If using dual forwarding mode, add these additional variables:

INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED=true
INSIGHTS_AGENT_ADDITIONAL_ENDPOINTS='{"https://app.datadoghq.com":["your-api-key"]}'
INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS='[{"api_key":"your-api-key","host":"agent-http-intake.logs.datadoghq.com","use_compression":true,"compression_level":2}]'

Mounting the OTel Configuration in Podman

You must mount the OTel configuration secret into the Insights Agent container. Add the appropriate mount flags to your podman run command:

For third-party only mode
--secret "{NAME_PREFIX}-otel-config,type=mount,target=/etc/datadog-agent/otel-config.yaml,mode=0644"
For dual forwarding mode (adds both secrets)
--secret "{NAME_PREFIX}-otel-config,type=mount,target=/etc/datadog-agent/otel-config.yaml,mode=0644" \
--secret "{NAME_PREFIX}-logs-config,type=mount,target=/etc/datadog-agent/conf.d/solace.d/logs.yml"

Replace {NAME_PREFIX} with the prefix you used when creating the secrets.

Example of Podman Run Command with Insights Forwarding Enabled

For appliance event brokers with Insights forwarding enabled, the --memory value should be set to 5888m to accommodate the appliance event broker class size. For more information, see Resource Requirements.

The following example runs the Insights Agent with Insights forwarding in "third-party only" mode on an appliance event broker. You can use this example instead of the line-by-line variable specification for installing the Insights Agent without forwarding.

As the insights Linux shell user, run the following command:

podman run \
  -d \
  --restart=always \
  --name="my-service-primary-insights-container" \
  --network host \
  # Core forwarding environment variables
  -e INSIGHTS_AGENT_THIRD_PARTY_FORWARDING_ENABLED="true" \
  -e INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED="false" \
  -e INSIGHTS_AGENT_BROKER_SIZE="APPLIANCE" \
  # Standard environment variables from Cloud Console
  -e INSIGHTS_AGENT_SITE="{INSIGHTS_AGENT_SITE}" \
  -e INSIGHTS_AGENT_TAGS="{INSIGHTS_AGENT_TAGS}" \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME="127.0.0.1" \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE="my-service-primary" \
  -e INSIGHTS_AGENT_NODE_ROLE="primary" \
  -e INSIGHTS_AGENT_SEMP_USERNAME="insights-agent" \
  -e INSIGHTS_AGENT_SEMP_PROTOCOL="http" \
  -e INSIGHTS_AGENT_SEMP_PORT="80" \
  -e INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION="true" \
  # Secrets for SEMP password, API key, and OTel config
  --secret "insights-agent-semp-password,type=env,target=INSIGHTS_AGENT_SEMP_PASSWORD" \
  --secret "insights-agent-api-key,type=env,target=INSIGHTS_AGENT_API_KEY" \
  --secret "my-service-primary-otel-config,type=mount,target=/etc/datadog-agent/otel-config.yaml,mode=0644" \
  # For dual forwarding mode, change INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED to "true" and add:
  # --secret "my-service-primary-additional-endpoints,type=env,target=INSIGHTS_AGENT_ADDITIONAL_ENDPOINTS" \
  # --secret "my-service-primary-logs-config-additional-endpoints,type=env,target=INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS" \
  # Mount broker logs (read-only)
  --mount type=bind,source=/usr/sw/jail/logs/,destination=/jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended
  --mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run \
  # Resource limits for appliances
  --cpus="2.2" \
  --memory="5888m" \
  --cpuset-cpus=4 \
  # Agent image
  gcr.io/gcp-maas-prod/solace-insights-agent:5d35eea

Validating the Insights Agent Installation

After configuring the Insights Agent, you can validate the Insights Agent is working by logging into your Insights Datadog account and viewing the Insights Agent host in the Datadog infrastructure list.

If you have an configured Insights forwarding to a third party observability platform, you should also validate that your Insights metrics and logs are reaching that platform.

To view the Insights Agent host in the Datadog infrastructure list, do the following:

  1. Log in to the Solace Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the Solace Cloud Console.
  2. Select Insights on the navigation bar.
  3. Click View in Datadog from any of the available tiles where it appears.
  4. If prompted to do so, log in to Datadog.
  5. From the menu on the left, select Infrastructure and then click Hosts.
  6. In the Search by field, enter one of the following values to filter the list to show only your Insights Agent host:
    • {INSIGHTS_AGENT_BROKER_HOSTNAME} where {INSIGHTS_AGENT_BROKER_HOSTNAME} is the name of the Insights Agent.
    • {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} where {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} is the name of the Insights Agent if you set this environment variable.
  7. Select the host to access its information. From this screen you can:
    1. click the Host Info tab to confirm the {INSIGHTS_AGENT_TAGS} are present.
    2. click the Logs tab to view the logs the host is ingesting. Logs are only created if actions occur on the software event broker.

Validating Insights Forwarding

After deploying with Insights forwarding enabled, verify that metrics and logs are reaching your third-party platform:

  1. As the insights user, check the Insights Agent container logs to verify the OTel collector started successfully:

    podman logs my-service-primary-insights-container

    Look for log entries indicating the solace-otel-agent process has started. If you see the message "third party forwarding is not configured, falling back silently to previous datadog-only telemetry", the OTel configuration is invalid or missing.

  2. Verify environment variables are set correctly using the process environment:

    strings /proc/$(pgrep -f datadog-agent)/environ | grep INSIGHTS

    Confirm that INSIGHTS_AGENT_FORWARDING_ENABLED=true and INSIGHTS_AGENT_OTEL_CONFIG_FILE is set to the correct path.

  3. Verify the Insights Agent health check is responding:

    curl http://localhost:5550/health

  4. Log into your third-party observability platform and verify metrics are appearing:

    • Look for metrics with the prefix solace. or pubsubplus.

    • Check for the datadog.agent.running heartbeat metric

    • Verify your appliance hostname (from INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE) appears in the metric metadata

If your Insights metrics and logs are not reaching your third-party observability platform, try the troubleshooting steps below.

Troubleshooting Insights Forwarding

You can try the following commands if your Insights metrics or logs do not appear in your third-party observability platform:

  • Verify the otel-config.yaml file exists on the appliance: 

    ls -l /usr/sw/insights/otel-config.yaml

  • Verify the podman secret exists: 

    podman secret ls | grep otel-config

  • Verify the otel-config.yaml file is accessible in the container: 

    podman exec my-service-primary-insights-container cat /etc/datadog-agent/otel-config.yaml

  • Check the Insights Agent logs for OTel collector errors: 

    podman logs my-service-primary-insights-container | grep -i error

  • Verify environment variables are set correctly: 

    podman exec my-service-primary-insights-container env | grep INSIGHTS_AGENT

  • Verify network connectivity to your third-party platform: 

    curl -I https://your-observability-platform.com

  • Ensure the Insights Agent image version supports third-party forwarding: Verify you're using image version 5d35eea or newer.

  • Check CPU and memory allocation: Verify the container has enough memory allocated based on its class size.

    podman stats my-service-primary-insights-container

  • Set up ongoing health monitoring: To proactively detect Insights Agent health issues, configure monitoring for the datadog.agent.running heartbeat metric in your third-party observability platform. For more information, see Monitoring Insights Agent Health for Self-Managed Event Brokers.

Upgrading the Insights Agent

  • Upgrading the Insights Agents requires a restarting the podman container, resulting in a momentary loss of metric and log collection.

  • Solace recommends creating a downtime in your Datadog account before upgrading the Insights Agent, to prevent no data alerts while the Insights Agent is not running. You should scope this downtime to service_name for All Monitors. For more information, see Scheduling Downtime.

  • When upgrading the Insights Agent, you must use the same otel-config.yaml file and environment variables from your original installation. Do not regenerate the configuration unless you are also changing your third-party platform settings.

Solace notifies you when new versions of the Insights Agent become available. The notification provides tags required to pull the Insights Agent image from the Solace Container Registry. You should upgrade the Insights Agent when new versions become available.

To upgrade the Insights Agent, perform the following steps:

  1. When you receive a notification for a new Insights Agent version, do one of the following:

    • If you configured your deployment to pull images from the Solace Container Registry, you have access to the latest version when it becomes available.

    • If you configured to pull the image from a manually-updated registry, you must update that registry prior to updating the Insights Agent deployment.

    • If you are using a tar file from products, you must manually download the new tar file.

  2. Stop the existing Podman container.

    podman stop {INSIGHTS_AGENT_CONTAINER_NAME}

  3. Remove the existing Podman container.

    podman rm {INSIGHTS_AGENT_CONTAINER_NAME}

  4. Start a new Podman container using the same podman run command you used for the initial installation of the Insights Agent, ensuring to update the image tag only to the version of the image you just pulled from the repository. Leave all other environment variables unchanged.

Uninstalling the Insights Agent

Solace recommends creating a downtime in your Datadog account before uninstalling the Insights Agent, to prevent erroneous alerts while the Insights Agent is not running. This downtime should be scoped to service_name for All Monitors. For more information, see Scheduling Downtime.

You can uninstall the Insights Agent by stopping and removing the container you created on your appliance event broker during the Insights Agent installation process. You should also remove any volumes you created to persist the Insights Agent tailing state, and remove registry credentials.

Uninstalling the Insights Agent does not end your Insights subscription. To end your Insights subscription, contact Solace.

Run the following commands to uninstall a the Insights Agent:

  1. Stop the Insights Agent container:

    podman stop {INSIGHTS_AGENT_CONTAINER_NAME}

  2. Remove the Insights Agent container:

    podman rm {INSIGHTS_AGENT_CONTAINER_NAME}

  3. Remove the Insights API and SEMP secrets:

    podman secret rm {SEMP_PASSWORD_SECRETS}
podman secret rm {API_KEY_SECRETS}

  4. Optional: If you no longer require the registry credentials, delete the downloaded config.json file.