PubSub+ Insights for Self-Managed Appliances

Using Insights for self-managed appliance's is available as a Controlled Availability (CA) feature in PubSub+ Cloud. Contact Solace to evaluate if this option meets your use case and to get more information.

You can benefit from the observability features of PubSub+ Insights for your self-managed PubSub+ Appliance by installing an Insights Agent on your appliance. This feature is offered as controlled availability.

After installing and configuring the Insights Agent, you can access and use the numerous Insights dashboards, monitors, and metrics provided by Solace in your Datadog sub-organization. For more information, see PubSub+ Insights Advanced Monitoring.

Before deploying Insights for your appliance, review the Considerations for Deploying Insights for Appliances.

The process for installing the Insights Agent includes three steps:

1. Preparing the Appliance for Insights Agent Installation

2. Configuring and Installing the Insights Agent

3. Validating the Insights Agent Installation

If you are installing the Insights Agent on appliances in a high availability (HA) configuration, you must repeat this process for each appliance in the HA pair using the environment variables specific to each node.

After installing the Insights Agent, you must periodically update it when new versions become available. Failure to update the Insights Agent may affect its ability to ingest your appliance's metrics and logs and provide them to Datadog. For more information, see Upgrading the Insights Agent.

You can uninstall the Insights Agent if required. For more information, see Uninstalling the Insights Agent.

Considerations for Deploying Insights for Appliances

Be aware of the following considerations when using Insights with a self-managed Appliance:

• Review the Considerations for Using Insights with Self-Managed Event Brokers.

• Your appliance must be running version soltr_10.25.3.2742 or later with Podman version 5.4.0 or later installed. For more information, see Changing Appliance Software.

• The Insights Agent runs containerized service using Podman on the appliance with the insights user account.

• You must configure a dedicated Solace CLI user with global read permissions that uses basic authentication on the appliance. For more information, see Configuring a SEMP User for the Insights Agent.

• You must configure your appliance before installing the Insights Agent. Preparing your appliance includes stopping the Solace Geneos Agent, if running. For more information, see Preparing the Appliance for Insights Agent Installation

• The Insights Agent connects to the appliance using HTTP on localhost (127.0.0.1) for SEMP requests.

• Your appliance must have DNS configured correctly to reach gcr.io and your Datadog site (e.g., datadoghq.com or datadoghq.eu).

• You must use the Add Insights Monitoring form in the Cloud Console to acquire the image pull secret and API key referenced in the CLI commands in the procedures in this document. For more information, see Insights for Self-Managed Event Brokers.

• Solace recommends securely saving the commands used in the procedures below on a per-installation basis. You will require them in the future when upgrading the Insights Agent.

Insights Agent Metric Collection Limitations on Appliances

While the Insights Agent provides the ability to collect most logs and metrics from your appliance, it cannot collect the following metrics for the default Message VPN:

• derived-metrics.client.connection.authentication.client-cert-auth.enabled

• derived-metrics.client.connection.authentication.internal.enabled

• derived-metrics.client.connection.authentication.ldap.enabled

• derived-metrics.client.connection.authentication.kerberos-auth.enabled

• derived-metrics.client.connection.authentication.oauth.enabled

Insights Agent Resource Requirements

The Insights Agent requires the following resources from your appliance:

Resource Type	Minimum Requirement
CPU	0.5 CPU cores
Memory	512 MiB
CPU Set	CPU core 4 (isolated)

Preparing the Appliance for Insights Agent Installation

Before installing the Insights Agent on your appliance, you must perform several steps to ensure you appliance is ready. These steps include:

1. Configuring a SEMP User for the Insights Agent

2. Accessing the appliance Linux shell and:

   1. Stopping the Solace Geneos Agent

   2. Switching to the Insights User

3. PubSub+ Insights for Self-Managed Appliances

4. PubSub+ Insights for Self-Managed Appliances

If you are installing the Insights Agent on appliances in a high availability (HA) configuration, you must repeat this process for each appliance in the HA pair.

Configuring a SEMP User for the Insights Agent

You must create a SEMP user for the Insights Agent so it can send SEMP requests. The user should be provided with the minimal required permissions to monitor all necessary metrics on the appliance, in this case, global read only permissions. For more information see CLI User Access Levels.

Log into the Solace CLI on your appliance and use the following CLI command to create the SEMP user on your appliance:

solace# enable
solace# configure
solace(configure)# create username insights-agent password <password> cli global-access-level read-only
solace(configure/username)# message-vpn default-access-level read-only
solace(configure/username)# end

Accessing the Appliance Linux Shell

After creating the SEMP user on the appliance, you must access the Linux shell on the appliance to continue configuring your appliance for installation of the Insights Agent.

From the Linux shell, you must:

1. stop the Solace Geneos Agent if it is running

2. switch to the Insights user

3. configure the Insights Agent download

For more information, see Accessing the Linux Shell Through Solace CLI.

Stopping the Solace Geneos Agent

You cannot run Solace Geneos Agent and the Insights Agent at the same time. If you are running Solace Geneos Agent on your appliance, you must stop it to prevent conflicts with the Insights Agent. Run the following Linux shell commands to stop the Solace Geneos Agent:

systemctl status solgeneos
systemctl stop solgeneos

Switching to the Insights User

You must switch to the insights user to continue to install the Insights Agent. Run the following commands to switch and verify that you are the Insights user:

su - insights
id  # Verify you are now the insights user

Configuring the Insights Agent Download

You have two choices for downloading the Insights Agent:

• Pulling the Insights Agent Image from the Solace Container Registry

• Downloading the Insights Agent from Products

Pulling the Insights Agent Image from the Solace Container Registry

You can configure your deployment to pull the image directly from the Solace Container Registry. You must use the registry secret provided by the Cloud Console (config.json) after completing the Add Insights Monitoring form to pull the image from the Solace Container Registry.

1. Log into the appliance Linux shell if you haven't done so and switch to the insights Linux shell user. For more information, see Accessing the Linux Shell Through Solace CLI, and Switching to the Insights User.

2. Place the config.json file to the /usr/sw/insights/.docker/config.json path on the appliance.

   scp <path_to_config_json>/config.json root@lab-128-82:/usr/sw/insights/.docker/config.json

3. You must set ownership of the config.json file to the insights user on the appliance. Use the following command to set ownership of the config.json file:

   chown insights:insights /usr/sw/insights/.docker/config.json

Use the directory where you saved the config.json file in step one with the --config flag in your podman commands that require pulling the image for installing the Insights Agent.

Downloading the Insights Agent from Products

If you don't want to provide access for your appliance to the Solace Container Registry, you can download the Insights Agent image tar files from products.solace.com. If you require assistance, contact Solace.

Configuring and Installing the Insights Agent

You will need to reuse the configuration files and podman run commands used in the following sections in the future when you upgrade the Insights Agent. Solace recommends securely saving these configuration files and commands on a per-installation basis, in a way that identifies the installation by container name so you can reuse them.

After preparing your appliance for installation of the Insights Agent, you can install the Insights Agent to the Podman container on the appliance. We provide two examples:

• Using an Environment File to Run the Insights Agent

• Using Line-by-Line Variable Specification to Run the Insights Agent

Running either of the commands produces the following results:

• Injects the environment variables required to configure the Insights Agent into its container.

• Shares the appliance's /usr/sw/jail/logs path with the Insights Agent under /jail/logs.

• Provides the container with recommended resources, as outlined in Insights Agent Resource Requirements.

• Pulls the latest version of the Insights Agent image from the Solace Container Registry.

You configure the Insights Agent using the environment variables generated by the Cloud Console after you compete the Add Insights Monitoring form. You can modify the rest of the commands to suit your requirements. We provide guidance for adding the environment variables and updating the commands below. For more information, see Configuring the Installation Command.

If you are installing the Insights Agent on appliances in a high availability (HA) configuration, you must repeat this process for each appliance in the HA pair, using the environment variables specific to each node.

Using an Environment File to Run the Insights Agent

To run the Insights Agent using an environment file, you must use the event broker properties generated by the Cloud Console after completing the Add Insights Monitoring form to create an environment file. As the insights Linux shell user, run the following command:

podman --config /path/to/config/json run \
  --name {INSIGHTS_AGENT_CONTAINER_NAME}     # Descriptive container name \
  -d                                         # Run in detached mode \
  --restart=always                           # Optional: Restart policy \
  --network host                             
  --env-file ./primary-node.env              # Load environment variables from file \
  --secret "insights-agent-semp-password,type=env,target={INSIGHTS_AGENT_SEMP_PASSWORD}" \
  --secret "insights-agent-api-key,type=env,target={INSIGHTS_AGENT_API_KEY}" \
  # Mount broker logs (read-only)
  --mount type=bind,source=/usr/sw/jail/logs/,destination=/jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended
  --mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" \
  --memory="512m" \
  --cpuset-cpus=4 \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Using Line-by-Line Variable Specification to Run the Insights Agent

To run the Insights Agent using a line-by-line variable specification, you must replace the environment variables (for example, {INSIGHTS_AGENT_TAGS}) in the code snippet below with the environment variables provided by the Cloud Console as KEY=VALUE pairs after completing the Add Insights Monitoring form as -e flags in the command. As the insights Linux shell user, run the following command:

podman --config /path/to/config/json run \
  - d \
  --restart=always \
  --name="my-primary-insights-container" # Descriptive container name \
  --network host \
  # Environment variables from Cloud Console \
  -e INSIGHTS_AGENT_SITE="{INSIGHTS_AGENT_SITE}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_TAGS="{INSIGHTS_AGENT_TAGS}" provided by the Cloud Console \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME="127.0.0.1" \
  -e INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE="my-service-primary"
  -e INSIGHTS_AGENT_NODE_ROLE="primary" \
  -e INSIGHTS_AGENT_SEMP_USERNAME="insights-agent" \  
  -e INSIGHTS_AGENT_SEMP_PROTOCOL="http" \
  -e INSIGHTS_AGENT_SEMP_PORT="80" \
  -e INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION="true" \
  --secret "insights-agent-semp-password,type=env,target="{INSIGHTS_AGENT_SEMP_PASSWORD}" \
  --secret "insights-agent-api-key,type=env,target="{INSIGHTS_AGENT_API_KEY}" \
  # Mount broker logs (read-only) \
  --mount type=bind,source=/usr/sw/jail/logs/,destination=/jail/logs,readonly \
  # Mount agent state volume (read-write) - Recommended \
  --mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run \
  # Recommended resource limits \
  --cpus="0.5" \
  --memory="512m" \
  --cpuset-cpus=4 \
  # Agent image - use specific version or 'latest' \
  gcr.io/gcp-maas-prod/solace-insights-agent:latest

Configuring the Installation Command

You can find guidance on configuring the environment variables and other portions of the commands above to meet the specific requirements of your environment in the sections below:

• Configuring Secrets

• Sharing the Log Files With the Insights Agent

• Collecting the Insights Agent Logs

• Naming the Insights Agent Container

• Persisting the Insights Agent’s Log Tailing State

Configuring Secrets

You must have pass installed and initialized on your appliance before completing the installation of the Insights Agent. For more information, see PubSub+ Insights for Self-Managed Appliances. With pass installed and initialized, you must configure the Insights API key and SEMP password as secretes in pass.

Use the following commands to create the secrets

podman secret create "insights-agent-semp-password"
podman secret create "insights-agent-api-key"

Use the following commands to populate the secrets with the values provided by completing the Add Insights Monitoring form in the Cloud Console.

--secret "insights-agent-semp-password,type=env,target=INSIGHTS_AGENT_SEMP_PASSWORD"
--secret "insights-agent-api-key,type=env,target=INSIGHTS_AGENT_API_KEY"

Sharing the Log Files With the Insights Agent

You must give the Insights Agent read-only access to your event brokers log directory, which you can do by mounting the event broker's log directory into the Insights Agent's container at /jail/logs. The direct path to the broker's log directory depends on the broker container setup on your appliance, common locations inside the broker container include:

• /var/lib/solace/jail/logs

• /usr/sw/jail/logs

Use the following command to the mount the appliance's log directory to the /jail/logs directory of the Insights Agent:

--mount type=bind,src={path to broker's log directory},target=/jail/logs,readonly

Collecting the Insights Agent Logs

You must enable log collection for the Insights Agent logs. To enable Insights Agent log collection set the INSIGHTS_AGENT_LOG_FORWARDING_ENABLED environment variable to 'true'.

Naming the Insights Agent Container

The Add Insights Monitoring form provides an IP address for the hostname of the Insights Agent container (127.0.0.1). You must use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable outlined in the Configuring Environment Variables table, to give a custom hostname to the Insights Agent container. The custom hostname appears as the display name for your appliance in Datadog, providing better identifiers for your Insights Agents.

Use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable generated by the Cloud Console after completing the Add Insights Monitoring form, with -insights-agent appended for clarity.

Persisting the Insights Agent’s Log Tailing State

You can prevent missing appliance logs caused by container restarts by configuring the Insights Agent to persist its log position markers in the /opt/datadog-agent/run directory.

To persist the Insights Agent's log tailing state, run the following command:

--mount type=bind,source=/usr/sw/insights/run,target=/opt/datadog-agent/run

Configuring Environment Variables

You must configure the environment variables using the environment variables generated by the Cloud Console after you complete the Add Insights Monitoring form.

If you are Using an Environment File to Run the Insights Agent you must create an environment file using these values.

The following table lists the environment variables, specifies if the variable is required, how the value is provided (by the Cloud Console or by you, the customer), the purpose of variable, and any additional important information.

Environment Key Variable	Required	Function	Notes, Limitations, and Examples
INSIGHTS_AGENT_API_KEY	Yes	Defines the credentials used to authenticate when connecting to the central monitoring service (Datadog) to submit event broker metrics and logs. Provided by the Cloud Console.	N/A
INSIGHTS_AGENT_SITE	Yes	Defines the central monitoring service (Datadog) region endpoint where Insights ingests event broker metrics and logs. Provided by the Cloud Console.	N/A
INSIGHTS_AGENT_TAGS	Yes	Metadata attached to your event broker's metrics and logs. These metadata tags uniquely identify the broker node within Insights and associate it with your organization, environment, and service. Provided by the Cloud Console.	N/A
INSIGHTS_AGENT_BROKER_HOSTNAME	Yes	The hostname or IP address that the Insights Agent uses to connect to the event broker node's SEMP and Health Check APIs. For appliance's the value is always 127.0.0.1	This variable's value also represents the name for the event broker in your Insights Datadog sub-organization. For appliance-based deployments, this value must be 127.0.0.1. You can use the INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE variable to use a custom name in Datadog.
INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE	Yes	Specified the custom name for the host broker node in Datadog. The value provided for this variable overrides the value in INSIGHTS_AGENT_BROKER_HOSTNAME. Provided by you, the customer.	Custom names you provide for the host broker node must meet the following requirements:  Have a maximum length of 255 characters. Must be RFC 1123 compliant. Examples: my-event-broker-service-primary k8s-node-name-broker-backup
INSIGHTS_AGENT_SEMP_PROTOCOL	Yes	Defines the protocol (HTTP or HTTPS) used by the Insights Agent to make SEMP requests to the event broker. Provided by the Cloud Console.	Possible values are http or https.
INSIGHTS_AGENT_SEMP_PORT	Yes	Defines the SEMP port of the event broker used by the Insights Agent. Provided by the Cloud Console.	Examples could include 8080 or 1943
INSIGHTS_AGENT_SEMP_USERNAME	Yes	Defines the read-only management user name used by the Insights Agent when making SEMP requests. Provided by the Cloud Console.	Appliance example: sempread
INSIGHTS_AGENT_SEMP_PASSWORD	Yes	Defines the password for the read-only management user name defined in the INSIGHTS_AGENT_SEMP_USERNAME variable. Provided by the Cloud Console.	Appliance example: sempreadpassword
INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION	Yes	The Insights Agent must always skip TLS cuertificate validation when making SEMP requests over HTTPS. The Insights Agent uses 127.0.0.1 as the hostname to send SEMP requests to the event broker. The event broker's certificate is issued for a hostname other than localhost, which will cause a validation mismatch if TLS verification is not disabled. Provided by the Cloud Console.	The value must be true.
INSIGHTS_AGENT_NODE_ROLE	Yes	Defines the role of the event broker node (primary, backup, or monitoring) that the Insights Agent is monitoring. Provided by the Cloud Console.	Possible values: primary or 0 backup or 1 monitoring or 2

Validating the Insights Agent Installation

After configuring the Insights Agent, you can validate the Insights Agent is working by logging into your Insights Datadog account and viewing the Insights Agent host in the Datadog infrastructure list.

To view the Insights Agent host in the Datadog infrastructure list, do the following:

1. Log in to the PubSub+ Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the PubSub+ Cloud Console.
2. Select Insights on the navigation bar.
3. Click View in Datadog from any of the available tiles where it appears.
4. If prompted to do so, log in to Datadog.
5. From the menu on the left, select Infrastructure and then click Hosts.
6. In the Search by field, enter one of the following values to filter the list to show only your Insights Agent host:
   • {INSIGHTS_AGENT_BROKER_HOSTNAME} where {INSIGHTS_AGENT_BROKER_HOSTNAME} is the name of the Insights Agent.
   • {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} where {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} is the name of the Insights Agent if you set this environment variable.
7. Select the host to access its information. From this screen you can:
   1. click the Host Info tab to confirm the {INSIGHTS_AGENT_TAGS} are present.
   2. click the Logs tab to view the logs the host is ingesting. Logs are only created if actions occur on the software event broker.

Upgrading the Insights Agent

Solace notifies you when new versions of the Insights Agent become available. The notification provides tags required to pull the Insights Agent image from the Solace Container Registry. You should upgrade the Insights Agent when new versions become available.

To upgrade the Insights Agent, perform the following steps:

1. When you receive a notification for a new Insights Agent version, do one of the following:

• If you configured your deployment to pull images from the Solace Container Registry, you have access to the latest version when it becomes available.

• If you configured to pull the image from a manually-updated registry, you must update that registry prior to updating the Insights Agent deployment.

• If you are using a tar file from products, you must manually download the new tar file.

2. Stop the existing Podman container.

   podman stop {INSIGHTS_AGENT_CONTAINER_NAME}

3. Remove the existing Podman container.

   podman rm {INSIGHTS_AGENT_CONTAINER_NAME}

4. Start a new Podman container using the same podman run command you used for the initial installation of the Insights Agent, ensuring to update the image tag only to the version of the image you just pulled from the repository. Leave all other environment variables unchanged.

Uninstalling the Insights Agent

You can uninstall the Insights Agent by stopping and removing the container you created on your appliance during the Insights Agent installation process. You should also remove any volumes you created to persist the Insights Agent tailing state, and remove registry credentials.

Uninstalling the Insights Agent does not end your Insights subscription. To end your Insights subscription, contact Solace.

Run the following commands to uninstall a the Insights Agent:

1. Stop the Insights Agent container:

   podman stop {INSIGHTS_AGENT_CONTAINER_NAME}
   

2. Remove the Insights Agent container:

   podman rm {INSIGHTS_AGENT_CONTAINER_NAME}

3. Remove the Insights API and SEMP secrets:

   podman secret rm {SEMP_PASSWORD_SECRETS}
   podman secret rm {API_KEY_SECRETS}
   

4. Optional: If you no longer require the registry credentials, delete the downloaded config.json file.