PubSub+ Insights for Self-Managed Kubernetes-Based Software Event Brokers

You can benefit from the observability features of PubSub+ Insights for your self-managed PubSub+ software event brokers by installing an Insights Agent in your Kubernetes cluster.

After installing and configuring the Insights Agent you can access and use the numerous Insights dashboards, monitors, and metrics provided by Solace in your Datadog sub-organization. For more information, see PubSub+ Insights Advanced Monitoring.

Before deploying Insights for your Kubernetes-based software event broker, review the Considerations for Deploying Insights for Kubernetes-Based Software Event Broker and the Insights Agent Resource Requirements.

The process for installing the Insights Agent includes three steps:

Downloading the Insights Agent
Installing the Insights Agent
Validating the Insights Agent Installation

After installing the Insights Agent, you must periodically update it when new versions become available. Failure to update the Insights Agent may affect its ability to ingest your software event broker metrics and log and provide them to Datadog. For more information, see Upgrading the Insights Agent.

You can uninstall the Insights Agent if required. For more information, see Uninstalling the Insights Agent.

Considerations for Deploying Insights for Kubernetes-Based Software Event Broker

Be aware of the following considerations when using Insights with a self-managed software event broker in a Kubernetes cluster:

Review the Considerations for Using Insights with Self-Managed Software Event Brokers.
You must use the Add Insights Monitoring form in the Cloud Console to acquire the image pull secret and environment variables used to configure the Insights Agent used in the procedures in this document.
Installation using Kubernetes Helm Quick Start requires Helm chart version 3.6.0 or later.

Insights Agent Resource Requirements

Each Insights Agent has the following resource requirements when deployed to a Kubernetes cluster:

Type	Request	Limit
CPU	200m	200m
Memory	256MiB	512MiB

Downloading the Insights Agent

You have two choices for downloading the Insights Agent:

Downloading the Insights Agent from the Solace Container Registry
Downloading the Insights Agent from Products

Downloading the Insights Agent from the Solace Container Registry

You can configure your deployment to pull the image directly from the Solace Container Registry, or from your own pull registry. You require the image pull secret provided by completing the Add Insights Monitoring form in the Cloud Console to pull the image from the Solace Container Registry.

To configure your deployment to pull from the Solace Container Registry, use the following command to apply the secret to the namespace where the software event broker is deployed

kubectl apply -f {SECRET}.yaml {BROKER-NAMESPACE}

Where:

{SECRET} is the the pull secret YAML provided by the Cloud Console after the Add Insights Monitoring form.
{BROKER-NAMESPACE} is the namespace where your software event broker is deployed.

Downloading the Insights Agent from Products

You can download the Insights Agent image tar files from products.solace.com. If you require assistance, contact Solace.

Installing the Insights Agent

You install the Insights Agent in its own container, on a per-software event broker node basis. That is, one Insights Agent per Standalone software event broker and three Insights Agents for an HA software event broker. You have two choices for installing the Insights Agent:

Installing the Insights Agent Using the Solace Helm Quick Start
Installing and Configuring the Insights Agent Using Direct Kubernetes Manifests

Installing the Insights Agent Using the Solace Helm Quick Start

Solace offers an open-source Kubernetes Helm Quick Start project for launching a PubSub+ software event broker on GitHub. You can use this Quick Start to install the Insights Agent. For more information, see the PubSub+ Kubernetes Helm Quick Start repository.

Before launching the Kubernetes Helm Quick Start you require the following environment values provided by the Cloud Console after completing the Add Insights Monitoring form:

INSIGHTS_AGENT_API_KEY
INSIGHTS_AGENT_SITE
INSIGHTS_AGENT_TAGS

If using the Solace Helm Quick Start to install the Insights Agent in Kubernetes, you only need the INSIGHTS_AGENT_API_KEY, INSIGHTS_AGENT_SITE, and INSIGHTS_AGENT_TAGS entries generated by the Add Insights Monitoring form. After entering your event broker's Service Name and Service ID, you can add fake values (for example, 'Insights') into the other fields on the form to generate the values you require more quickly.

The Helm Quick Start handles all of the remaining environment variables based on your software event broker's configuration. You should consult the INSIGHTS.md documentation provided in the Kubernetes Helm Quick Start for guidance on configuring specific values file parameters, including:

insights.enabled=true
insights.environmentVariables.INSIGHTS_AGENT_API_KEY={INSIGHTS_AGENT_API_KEY} provided by the Cloud Console.
insights.environmentVariables.INSIGHTS_AGENT_SITE={INSIGHTS_AGENT_SITE} provided by the Cloud Console
insights.environmentVariables.INSIGHTS_AGENT_TAGS={INSIGHTS_AGENT_TAGS} provided by the Cloud Console (You only need to provide the base tag string provided by the Cloud Console; the Helm chart appends node-specific details).
(Optional) image.insights.pullSecretName={YOUR_SECRET_NAME}. (Only required if you are using a custom image pull secret name, not gcr-reg-secret).

Once you have configured the specific values file parameters, install the the Insights Agent using the Quick Start Helm Chart.

Installing and Configuring the Insights Agent Using Direct Kubernetes Manifests

You can install the Insights Agent for your software event brokers using direct Kubernetes manifests in two steps:

Creating a Secret for the Insights Agent Environment Variables
Adding an Insights Agent Sidecar Container

Creating a Secret for the Insights Agent Environment Variables

Create a Kubernetes resource similar to the following example:

apiVersion: v1
kind: Secret
metadata:
  name: insights-agent-env-vars     # use a unique name
  metadata: {YOUR-BROKER-NAMESPACE}
type: Opaque
stringData:
  INSIGHTS_AGENT_API_KEY: "{INSIGHTS_AGENT_API_KEY}"
  INSIGHTS_AGENT_SITE: "{INSIGHTS_AGENT_SITE}"
  INSIGHTS_AGENT_TAGS: "{INSIGHTS_AGENT_TAGS}"
  INSIGHTS_AGENT_SEMP_USERNAME: "{INSIGHTS_AGENT_SEMP_USERNAME}"
  INSIGHTS_AGENT_SEMP_PASSWORD: "{INSIGHTS_AGENT_SEMP_PASSWORD}"
  INSIGHTS_AGENT_SEMP_PROTOCOL: "{INSIGHTS_AGENT_SEMP_PROTOCOL}"                         # for example, http or https
  INSIGHTS_AGENT_SEMP_PORT: "{INSIGHTS_AGENT_SEMP_PORT}"                                 # for example, "8080"
  INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION: "{INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION}"   # for example, "true" or "false"
  INSIGHTS_AGENT_HEALTH_CHECK_PORT: "{INSIGHTS_AGENT_HEALTH_CHECK_PORT}"                 # for example, "5550"
  
  # == Required for Sidecar Pattern ==
  # Insights Agent connects to broker within the same Pod via localhost
  INSIGHTS_AGENT_BROKER_HOSTNAME: "localhost"
  
  # == Node Role - Varies by Deployment Type ==
  #
  # Option A: For *Single-Replica* workloads (Standalone broker, or separate Deployments per HA node)
  # Uncomment and set the correct role for THIS specific deployment's secret.
  # INSIGHTS_AGENT_NODE_ROLE: "primary" # Or "backup", "monitoring", "0", "1", "2"
  #
  # Option B: For *Multi-Replica StatefulSets* (Recommended for HA)
  # OMIT INSIGHTS_AGENT_NODE_ROLE here. It will be injected dynamically in the Pod spec later.
 
  # Add any other environment variables provided by the Cloud Console

You must replace the environment variables under stringData in the Kubernetes resource with values outlined in the following table. You are provided with values for most of the variables by the Cloud Console after you complete the Add Insights Monitoring form.

The following table lists the environment variables, specifies if the variable is required, how the value is provided (by the Cloud Console or by you, the customer), the purpose of variable, and any additional important information.

Environment Key Variable	Required	Function	Notes, Limitations, and Examples
`INSIGHTS_AGENT_API_KEY`	Yes	Defines the credentials used to authenticate when connecting to the central monitoring service (Datadog) to submit event broker metrics and logs. Provided by the Cloud Console.	N/A
`INSIGHTS_AGENT_SITE`	Yes	Defines the central monitoring service (Datadog) region endpoint where Insights ingests event broker metrics and logs. Provided by the Cloud Console.	N/A
`INSIGHTS_AGENT_TAGS`	Yes	Metadata attached to your event broker's metrics and logs. These metadata tags uniquely identify the broker node within Insights and associate it with your organization, environment, and service. Provided by the Cloud Console.	N/A
`INSIGHTS_AGENT_BROKER_HOSTNAME`	Yes	The hostname or IP address that the Insights Agent uses to connect to the event broker node's SEMP and Health Check APIs. Provided by the Cloud Console.	This variable's value also represents the name for the event broker in your Insights Datadog sub-organization. For Kubernetes sidecar deployments, you must set this variable to `localhost`. For other delpoyment types, you can optionally use the `INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE` to use a custom name in Datadog. If you do not configure the `INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE` variable below, this variable must: have a maximum length of 255 characters. be RFC 1123 compliant. If you do configure the `INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE` variable below, this variable must be a valid DNS hostname, IP address, or container name. Examples: `my-event-broker-service-primary` `10.0.0.1` `<broker-container-name>` You must not use the following values unless you configure the `INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE` variable: `localhost` `localhost.localdomain` `localhost6.localdomain6` `ip6-localhost`
`INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE`	Docker/Podman: No Kubernetes: Yes	Specified the custom name for the host broker node in Datadog. The value provided for this variable overrides the value in `INSIGHTS_AGENT_BROKER_HOSTNAME`. Provided by you, the customer.	While this variable is not mandatory in containerized environments, it is useful for providing identifiers for your event brokers in Insights. Custom names you provide for the host broker node must meet the following requirements: Have a maximum length of 255 characters. Must be RFC 1123 compliant. Examples: `my-event-broker-service-primary` `k8s-node-name-broker-backup`
`INSIGHTS_AGENT_SEMP_PROTOCOL`	Yes	Defines the protocol (HTTP or HTTPS) used by the Insights Agent to make SEMP requests to the event broker. Provided by the Cloud Console.	Possible values are `http` or `https`.
`INSIGHTS_AGENT_SEMP_PORT`	Yes	Defines the SEMP port of the event broker used by the Insights Agent. Provided by the Cloud Console.	Examples could include `8080` or `1943`
`INSIGHTS_AGENT_SEMP_USERNAME`	Yes	Defines the read-only management user name used by the Insights Agent when making SEMP requests. Provided by the Cloud Console.	Example: `insights`
`INSIGHTS_AGENT_SEMP_PASSWORD`	Yes	Defines the password for the read-only management user name defined in the `INSIGHTS_AGENT_SEMP_USERNAME` variable. Provided by the Cloud Console.	Example: `password`
`INSIGHTS_AGENT_SEMP_SKIP_SSL_VALIDATION`	Yes	Defines whether the Insights Agent should skip TLS certificate verification when making SEMP requests over HTTPS. Provided by the Cloud Console.	You can set this variable to `true` if you use self-signed certificates, or don't require validation. The default value is `true`. Possible values are `true` or `false`.
`INSIGHTS_AGENT_HEALTH_CHECK_PORT`	Yes	Defines the plain-text HTTP health check port of the event broker node used by the Insights Agent. Provided by the Cloud Console.	Example: `5550`
`INSIGHTS_AGENT_NODE_ROLE`	Yes	Defines the role of the event broker node (primary, backup, or monitoring) that the Insights Agent is monitoring. Provided by the Cloud Console. Deployments in Kubernetes may need to adjust this variable's value when using a direct Kubernetes manifest (YAML file and multi-replica)	For Kubernetes sidecar deployments using: single-replica StatefulSets: You must set this value statically for each deployment (for example, `primary` for the primary node's secret or deployment). Do not inject it dynamically into the Pod spec. multiple-replica StatefulSets: You must inject this value dynamically into the Pod using `valueFrom` referencing a Pod label or field that indicates the replica index. Ensure the referenced label or field exists and provides the correct index corresponding to `primary (0)`, `backup (1)`, or `monitoring (2)` Possible values: `primary` or `0` `backup` or `1` `monitoring` or `2`

After creating the environment variable, apply it as a secret to your the namespace for your software event broker:

kubectl apply -f insights-agent-env-secret.yaml -n {YOUR-BROKER-NAMESPACE}

Where {YOUR-BROKER-NAMESPACE} is the namespace for your software event broker.

Adding an Insights Agent Sidecar Container

You must add the Insights Agent as a sidecar container to the software event broker pod. The configuration is similar across all resource types (StatefulSets, Deployments, etc.) and requires shared access to the software event broker's log files.

We provide an example code snippet below that you can use as a guide to modify the spec.template section of your software event broker's workload resource. You can adjust the container definition properties to meet your environment specifications, and you should specifically review the following:

Image pull secrets: Add a reference to the imagePullSecret. If you use the Solace Container Registry, you can use the following:
```
imagePullSecrets:
  - name: gcr-reg-secret
```
Volume mounts: Ensure Insights Agent has a volumeMount for its data and logs. For example, this could be one of the following:
```
mountPath: /var/lib/solace
mountPath: /usr/sw
```
Volume sharing: The software event broker container and Insights Agent sidecar must share the same underlying volume (for example, PersistentVolumeClaim, emptyDir, etc.). The Insights Agent mounts the software event broker's log directory (via subPath) in a read-only state at /jail/logs.
Persistent storage for log tailing state: The Insights Agent must be able to store log position markers to maintain its log position state between restarts. To allow the log position markers, mount a unique subPath (for example, opt/datadog-agent/run) from the same shared volume to the Insights Agent's /opt/datadog-agent/run directory with permissions that allow the Insights Agent's user to write to the directory.
Application continuity: Solace recommends setting the following value for any Kubernetes service that exposes the software event broker to ensure traffic continues to route to the software event broker during an Insights Agent upgrade:
```
publishNotReadyAddresses: true
```
Volume types:You should adapt the volumes and volumeMounts configurations based on your storage strategy (for example, volumeClaimTemplates in StatefulSets or existing PVCs in Deployments), however, it is critical that you provide shared access to the log location.
Configure environment variables: Ensure that you configure the environment variables correctly. Some variables must be set manually, while most can be pulled from the secret you created for the environment variables in Creating a Secret for the Insights Agent Environment Variables. The table in the previous section provides information about installation specific requirements, but the following three environment variables have specific requirements in a sidecar configuration:
- INSIGHTS_AGENT_BROKER_HOSTNAME: You must set this variable to localhost.
- INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE: You must specify this value in the Insights Agent container's environment variable specification as spec.nodeName.
- INSIGHTS_AGENT_NODE_ROLE:
  - single-replica StatefulSets: You must set this value statically for each deployment (for example, primary for the primary node's secret or deployment). Do not inject it dynamically into the Pod spec.
  - multiple-replica StatefulSets: You must inject this value dynamically into the Pod using valueFrom referencing a Pod label or field that indicates the replica index. Ensure the referenced label or field exists and provides the correct index corresponding to primary (0), backup (1), or monitoring (2).

spec:
  # ... other StatefulSet spec fields ...
  template:
    metadata:
      # ... pod metadata ...
    spec:
      # Reference the GCR secret (adjust name if needed)
      imagePullSecrets:
        - name: gcr-reg-secret
      containers:
        # === Your Existing Solace Broker Container ===
        - name: solace-broker # Or your broker container's name
          image: {YOUR_BROKER_IMAGE}
          # ... other broker container config ...
          volumeMounts:
            - name: data # Matches volume name below
               # Modern single mount point:
               mountPath: /var/lib/solace
               # OR Legacy mount point (requires subPath on the volume definition):
               # mountPath: /usr/sw/jail
               # subPath: jail
             # ... other broker volume mounts
        # === Insights Agent Sidecar Container ===
        - name: insights-agent
          image: gcr.io/gcp-maas-prod/solace-insights-agent:latest
          imagePullPolicy: Always
          envFrom:
            # Load common environment variables from the Secret
            - secretRef:
                name: insights-agent-env-vars # Matches Secret name created earlier
          env:
            # === Required Overrides/Dynamic Values ===
            # Override hostname displayed in Datadog/Insights. Using Node Name is recommended.
            - name: {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE}
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # === Node Role Injection (for Multi-Replica StatefulSets ONLY) ===
            # If using a single StatefulSet for HA (replicas=3):
            # Adjust fieldPath if your index label is different (e.g., parse from metadata.name)
            - name: INSIGHTS_AGENT_NODE_ROLE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
          resources:
            # Recommended resource requests and limits
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi
          securityContext:
            # Recommended security settings
            # runAsUser: {UID} # Specify UID if needed, matching volume permissions
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          volumeMounts:
            # Mount broker logs (read-only)
            # 'subPath' should match the relative path to logs within the broker's mount point
            # e.g., if broker mounts 'data' to '/var/lib/solace', logs are under 'jail'
            - name: data # Matches volume name below
              mountPath: /jail
              subPath: jail # Relative path within the shared volume
              readOnly: true
            # Mount directory for agent state (read-write)
            - name: data # Matches volume name below
              mountPath: /opt/datadog-agent/run # Agent state directory
              subPath: opt/datadog-agent/run # Unique subPath on the shared volume
      # === Shared Volume Definition ===
      volumes:
        # This volume is shared between the broker and the agent sidecar.
        # For StatefulSets, this is often managed by volumeClaimTemplates.
        # For Deployments, it might be a pre-defined PVC.
        - name: data
          persistentVolumeClaim:
            # This claimName would typically be auto-generated by a StatefulSet's volumeClaimTemplates
            # Or refer to a specific PVC for Deployments.
            claimName: your-pvc-name-managed-by-sts-or-deployment # Adjust as per your setup
        # Example using emptyDir (suitable for single-node/testing ONLY, data lost on pod deletion)
        # - name: data
        #   emptyDir: {}
        # Example using hostPath (Use with caution, binds pod to specific node, security implications)
        # - name: data
        #   hostPath:
        #     path: /path/on/host/for/broker-data
        #     type: DirectoryOrCreate

Validating the Insights Agent Installation

After configuring the Insights Agent, you can validate the Insights Agent is working by logging into your Insights Datadog account and viewing the Insights Agent host in the Datadog infrastructure list.

To view the Insights Agent host in the Datadog infrastructure list, do the following:

Log in to the PubSub+ Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the PubSub+ Cloud Console.
Select Insights on the navigation bar.
Click View in Datadog from any of the available tiles where it appears.
If prompted to do so, log in to Datadog.
From the menu on the left, select Infrastructure and then click Hosts.
In the Search by field, enter one of the following values to filter the list to show only your Insights Agent host:
- {INSIGHTS_AGENT_BROKER_HOSTNAME} where {INSIGHTS_AGENT_BROKER_HOSTNAME} is the name of the Insights Agent.
- {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} where {INSIGHTS_AGENT_BROKER_HOSTNAME_OVERRIDE} is the name of the Insights Agent if you set this environment variable.
Select the host to access its information. From this screen you can:
1. click the Host Info tab to confirm the {INSIGHTS_AGENT_TAGS} are present.
2. click the Logs tab to view the logs the host is ingesting. Logs are only created if actions occur on the software event broker.

Upgrading the Insights Agent

Upgrading the Insights Agents requires a restart of your software event broker. You must consider the resulting Messaging Connectivity outage before upgrading the Insights Agent.
Solace recommends creating a downtime in your Datadog account before upgrading the Insights Agent, to prevent no data alerts while the Insights Agent is not running. You should scope this downtime to service_name for All Monitors. For more information, see Scheduling Downtime.

Solace notifies you when new versions of the Insights Agent become available. The notification provides tags required to pull the Insights Agent image from the Solace Container Registry. You should upgrade the Insights Agent when new versions become available.

To upgrade the Insights Agent, perform the following steps:

When you receive a notification for a new Insights Agent version, do one of the following:

If you configured your deployment to pull images from the Solace Container Registry, you have access to the latest version when it becomes available.
If you configured to pull the image from a manually-updated registry, you must update that registry prior to updating the Insights Agent deployment.
If you are using a tar file from products, you must manually download the new tar file.

If your are pulling the Insights Agent images, the pull policy for the Insights Agent container is set to Always, causing it to pull a new image when restarted. Restart the Insights Agent container in each Pod with the following command:
```
kubectl exec {BROKER_POD_NAME} -n {YOUR-BROKER-NAMESPACE} -c insights-agent -- kill 1
```

Uninstalling the Insights Agent

Uninstalling the Insights Agents requires a restart of your software event broker. You must consider the resulting Messaging Connectivity outage before uninstalling the Insights Agent.

Solace recommends creating a downtime in your Datadog account before uninstalling the Insights Agent, to prevent erroneous alerts while the Insights Agent is not running. This downtime should be scoped to service_name for All Monitors. For more information, see Scheduling Downtime.

The method for uninstalling the Insights Agent depends on how you installed it:

Uninstalling an Insights Agent Installed by Helm Quickstart
Uninstalling an Insights Agent Installed Using Direct Kubernetes Manifests

Uninstalling the Insights Agent does not end your Insights subscription. To end your Insights subscription, contact Solace.

Uninstalling an Insights Agent Installed by Helm Quickstart

To uninstall an Insights Agent deployed using the Helm Quickstart installation method, perform a helm upgrade. Set insights.enabled to false and clear the environment variables you set during the installation in the upgrade command, as shown in the following example:

helm upgrade <release_name> solacecharts/pubsubplus --version <chart_version> \
  --set insights.enabled=false \
  --set insights.environmentVariables.INSIGHTS_AGENT_API_KEY="" \
  --set insights.environmentVariables.INSIGHTS_AGENT_SITE="" \
  --set insights.environmentVariables.INSIGHTS_AGENT_TAGS="" \
  --reuse-values \
  -n <your-broker-namespace>

Optional: If no other workloads in the namespace require the gcr-reg-secret image pull secret, you can delete it with the following command:

kubectl delete secret gcr-reg-secret -n <your-broker-namespace>

Uninstalling an Insights Agent Installed Using Direct Kubernetes Manifests

To uninstall an Insights Agent installed using direct Kubernetes manifests, you must generally perform the installation steps in reverse. While every Kubernetes environment is unique, be sure to:

Edit your software event broker's Deployment or StatefulSet YAML file.
1. Remove the insights-agent container definition from the spec.template.spec.containers list.
2. Remove the volumeMounts specific to the Insights Agent (for example, /jail/logs and /opt/datadog-agent/run).
3. (Optional) Remove the imagePullSecrets field if the software event broker container no longer needs the gcr-reg-secret.
Apply the above changes to the Deployment or SatefulSet YAML file with the following command:
```
kubectl apply -f your-workload-definition.yaml -n <your-broker-namespace>
```

Delete the Insights Agent Environment Secret:

kubectl delete secret insights-agent-env-vars -n <your-broker-namespace> # Use the actual Secret name

(Optional) If the gcr-reg-secret Image Pull Secret is no long needed, you can delete it with the following command:
```
kubectl delete secret gcr-reg-secret -n <your-broker-namespace>
```
(Optional) Clean up persistent volume SubPath: If you used a subPath on the shared persistent volume for the Insights Agent state (for example, opt/datadog-agent/run in the example provided above), the data will remain on the persistent volume. If you want to remove this directory, you may need to manually access the persistent volume's filesystem.

Provide feedback