Forwarding Solace Insights Metrics and Logs for Self-Managed Event Brokers

If you're using Solace Cloud and want to forward your event broker service metrics and logs to a third-party observability platform, see Forwarding Solace Insights Metrics and Logs.

You can forward Solace Insights metrics and logs from your self-managed software and appliance event brokers to third-party observability platforms of your choice, including Dynatrace, New Relic, Splunk, or any other platform that supports the OpenTelemetry Protocol (OTLP). You can send your data to your Insights Datadog account, to a third-party observability platform using an OpenTelemetry (OTel) collector, or to both simultaneously.

Forwarding your Insights metrics and logs centralizes the Insights data alongside any other data you collect from your infrastructure or systems. Having your data centralized in one observability platform allows your application teams to make sense of it with tools they are already familiar with.

For an overview of Insights forwarding with self-managed event brokers, see Understanding How Forwarding of Insights Metrics and Logs Works for Self-Managed Event Brokers.

Before configuring Insights forwarding for self-managed event brokers, review the Considerations for Insights Forwarding of Metrics and Logs.

When forwarding metrics and logs, you must provide the required resources to the OTel collector, and select a forwarding mode.

Configuring and deploying Insights forwarding for self-managed event brokers is similar to deploying Insights without forwarding. For more information, see Configuring Forwarding of Insights Metrics and Logs for Self-Managed Event Brokers.

When deploying the Insights forwarding you must configure your network connectivity to allow Insights components to talk to each other. There are additional configurations required if you use an HTTP/HTTPS proxy, or deploy to an air-gapped environment.

After deploying Insights forwarding, you should validate the installation, and then monitor the health of your Insights Agent to ensure your metrics and logs are continuously forwarded.

Solace provides some troubleshooting steps if you are having issues with your Insights forwarding deployment.

When new Insights Agent versions become available, you can upgrade the Insights Agent.

Understanding How Forwarding of Insights Metrics and Logs Works for Self-Managed Event Brokers

Insights forwarding uses an OTel collector that runs alongside the Insights Agent as a sidecar process.

  1. The Insights Agent collects metrics from your self-managed event broker via SEMP and ingests event logs.

  2. When Insights forwarding is enabled, the Insights Agent sends data it collects to the OTel collector instead of directly to Datadog.

  3. The OTel collector processes and forwards the metrics and logs according to the forwarding mode you've configured:

    • Third-party only mode: Metrics and logs are sent exclusively to your specified third-party observability platform.

    • Dual forwarding mode: Metrics and logs are sent to both Datadog and your specified third-party observability platform.

    For more information, see Insights Forwarding Modes for Self-Managed Event Brokers.

If you have entitlements to Insights with forwarding, the OTel collector configuration is generated when you complete the Add Insights Monitoring page in the Cloud Console. When using Insights forwarding, the downloadable package generated by the Add Insights Monitoring page includes additional files:

  • otel-config.yaml — OTel collector configuration with your third-party platform endpoint and credentials

  • logs.yaml — Datadog agent log collection configuration (required for dual forwarding mode only; omit when using third-party only mode)

  • Environment variables — Configuration parameters for the Insights Agent

  • config.json — Registry secret for pulling the Insights Agent image

Considerations for Insights Forwarding of Metrics and Logs

Be aware of the following considerations when using third-party forwarding for self-managed event brokers:

  • You must have an active Insights subscription with an entitlement to Insights forwarding to use Insights Forwarding. To get Insights, contact Solace.

  • Review the Considerations for Using Insights with Self-Managed Event Brokers.

  • Third-party forwarding is only available for software event brokers using container-based deployments (Kubernetes/Helm, Docker, Podman). It is not supported for software event brokers in VM-based deployments.

  • The OTel collector runs as a sidecar process within the Insights Agent container and requires additional resources. See Resource Requirements.

  • You must create a forwarding destination using the Cloud Console before you generate your Insights Agent secret and environment properties using the Add Insights Monitoring page in the Cloud Console.

  • Configuration is generated via the Add Insights Monitoring page in the Cloud Console, which allows you to download the generated variables required by the Insights Agent.

  • You must provide your Global Access Read-Only Management User credentials during configuration. Solace does not store these credentials. The credentials are embedded in the generated variable file for the Insights Agent. It is your responsibility to secure the variable file.

  • You are responsible for ensuring that your chosen observability platform meets any data residency or compliance requirements.

  • When using "third-party only" mode (no Datadog), all metrics and logs are sent exclusively to your third-party platform. No data, including agent error logs, is sent to Datadog.

  • When using "Insights Datadog platform only" mode (no third-party), all metrics and logs are sent exclusively to your Insights Datadog account. The forwarding configuration is generated as if third-party forwarding is disabled.

  • For air-gapped or offline environments, offline configuration templates are available. Contact Solace for details.

  • Solace provides the Insights metrics and logs as is. It is your responsibility to configure your observability platform to use the data.

Resource Requirements

When enabling Insights forwarding, you must provide additional resources for the OTel collector beyond those required by the standard Insights Agent in your deployment scenario:

You must account for both additional compute and memory resource requirements:

Compute Resource Requirements by Deployment Type for Third-Party Forwarding

The following table provides the additional compute resource requirements you must provide the Insights Agent and OTel collector. The compute requirements differ by deployment type, but are the same regardless of event broker class within a given deployment type (Kubernetes/Helm, Docker/Podman, and Appliance).

Deployment Type Standard Mode (Datadog Only) Third-Party Forwarding Mode
Software event broker deployed in Kubernetes/Helm 256/512 MiB memory
200m CPU		 See table below for broker-size-specific memory requirements
Software event broker deployed in Docker/Podman 512 MiB memory
0.5 CPU cores		 See table below for broker-size-specific memory requirements
Appliance event broker 512 MiB memory
0.5 CPU cores		 See table below for broker-size-specific memory requirements

Memory Requirements by Event Broker Class Size for Third-Party Forwarding

The following table provides the additional memory resource requirements you must provide the Insights Agent and OTel collector. The memory resource requirements depend on your event broker's class, but are the same for an event broker class across all deployment types:

Broker Size Class Minimum Memory Required
STD-100-Dev 768 MiB

STD-1K -Dev

ENT-1K

 1408 MiB
ENT-10K 2432 MiB
ENT-100K 4480 MiB
ENT-200K 5888 MiB

Insights Forwarding Modes for Self-Managed Event Brokers

Insights forwarding of metrics and logs for software or appliance event brokers supports three modes of operations:

Datadog Only (Standard Mode)

This is the standard mode when you use Insights for self-managed event brokers without forwarding. Your Insights metrics and logs are sent exclusively to your Insights Datadog account.

Third-Party Only Mode

Third-party only mode forwards all Insights metrics and logs exclusively to your chosen third-party observability platform. When you enable third-party only, no data is sent to your Insights Datadog account, including agent error logs. This mode is suitable for organizations that:

  • have standardized on a single observability platform

  • want to avoid dual data egress costs

  • have data residency or compliance requirements that prevent sending data to Datadog

Dual Forwarding Mode (Datadog + Third-Party)

Dual forwarding mode forwards all metrics and logs to both Datadog and your chosen third-party platform simultaneously. This mode allows you to:

  • maintain access to the Solace-provided Insights dashboards, monitors, and metrics in your Insights account in Datadog

  • centralize data alongside other infrastructure metrics in your preferred observability platform

  • gradually migrate to a third-party platform while maintaining Datadog access

Configuring Forwarding of Insights Metrics and Logs for Self-Managed Event Brokers

To forward your Insights metrics and logs for your software or appliance event broker, you must create a forwarding destination using the Cloud Console. After you create the forwarding destination, you use the Cloud Console to generate the required registry secrets, and environment variables for configuring the event broker.

To configure Insights forwarding, follow these steps:

  1. Log in to the Solace Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the Solace Cloud Console.

  2. Configure a forwarding destination in Solace Cloud using the Cloud Console.

  3. Complete the steps to generate registry secrets and environment properties for the Insights Agent, using the Cloud Console.

  4. Configure your self-managed event broker according to the instructions for the type of event broker and its deployment:

    Software Event Broker in Kubernetes

    For software event brokers running in Kubernetes, see Configuring Insights Forwarding for Self-Managed Kubernetes-Based Software Event Brokers .

    This deployment method uses Kubernetes Secrets to mount the otel-config.yaml file into the Insights Agent container.

    Software Event Broker in a Docker or Podman Container

    For software event brokers running in standalone Docker or Podman containers, see Configuring Insights Forwarding for Self-Managed Container-Based Software Event Broker.

    This deployment method uses either podman secrets (for Podman) or bind mounts (for Docker) to provide the otel-config.yaml file to the Insights Agent container.

    Appliance Event Broker

    For appliance event brokers, see Configuring Insights Forwarding for Self-Managed Appliance Event Brokers.

    This deployment method uses rootless Podman with podman secrets to run the Insights Agent as the insights user.

Validating Third-Party Forwarding

After deploying the Insights Agent with Insights forwarding enabled, you should verify that metrics and logs reach your chosen observability platform:

If metrics or logs are not appearing and the provided troubleshooting doesn't help, contact Solace.

Monitoring Insights Agent Health for Self-Managed Event Brokers

Solace has no visibility into your software event broker or appliance event broker after the Insights Agent is deployed. When you enable third-party forwarding, you must monitor the health of the Insights Agent to ensure your metrics and logs are collected and forwarded correctly. For more information, see:

Using the Insights Agent Heartbeat Metric

The Insights Agent emits a heartbeat metric named datadog.agent.running that serves as the primary health indicator. This metric is forwarded to the third-party observability platform you configured, along with your other event broker metrics and logs.

Heartbeat Metric Characteristics:

  • Metric Name: datadog.agent.running

  • Emission Frequency: Approximately every 60 seconds when the agent is healthy

  • Metric Value: 1 indicates the Insights Agent is running

  • Metric Tags: Includes ha_role and maas_ha_role attributes to distinguish between the primary, backup, and monitoring nodes in high-availability (HA) deployments

For more information, see Insights Agent Heartbeat (For Software and Appliance Event Brokers with Insights Forwarding).

Setting Up Health Monitoring

To monitor the health of the Insights Agent in your third-party observability platform, perform these steps:

  1. Verify the heartbeat metric is arriving: After deploying the Insights Agent, confirm that the datadog.agent.running metric appears in your third-party observability platform. The metric should appear within a few minutes of starting up the Insights Agent.

  2. Create an alert for missing heartbeats: Configure your third-party observability platform to alert you when the datadog.agent.running metric stops appearing. Solace recommends alerting when the metric has not been received for 5 minutes or more (approximately 5 consecutive missed heartbeats).

  3. Monitor by HA role (if applicable): For HA deployments, create separate monitors for each HA role (primary, backup, monitoring) using the ha_role or maas_ha_role tag to identify which node has stopped reporting.

  4. Verify broker metrics are appearing: In addition to the heartbeat metric, confirm that your event broker metrics (such as those with the solace. or pubsubplus. prefix) are being forwarded to your third-party observability platform.

Understanding Health Status

The following table provides patterns that indicate the health status of your Insights Agent:

Pattern Interpretation
Heartbeat metric appears regularly (every ~60 seconds) The Insights Agent is running and successfully forwarding metrics.
Heartbeat metric stops appearing The Insights Agent may have stopped, crashed, or lost connectivity to the third-party observability platform. Investigate Insights Agent container logs and verify network connectivity.
Heartbeat metric appears but event broker metrics do not The Insights Agent is running but may not be collecting metrics from the event broker. Verify SEMP connectivity between the Insights Agent and the event broker.
Heartbeat stops after Insights Agent upgrade The upgrade may have failed or the Insights Agent configuration may be incorrect. Verify that the otel-config.yaml file is still mounted and environment variables are set correctly.

Troubleshooting Missing Heartbeats

If the datadog.agent.running metric stops appearing in your third-party observability platform:

  1. Check agent container status: Verify the Insights Agent container is running using your container runtime (kubectl, docker ps, or podman ps).

  2. Review agent logs: Check the Insights Agent container logs for errors. The OTel collector logs to /etc/datadog-agent/output-logs.log and /etc/datadog-agent/error-output-logs.log within the container.

  3. Verify OTel configuration: Ensure the otel-config.yaml file is correctly mounted and readable by the Insights Agent. See the validation procedures for your deployment type.

  4. Check environment variables: Verify that forwarding environment variables are set correctly, especially INSIGHTS_AGENT_FORWARDING_ENABLED=:

    • true with third-party forwarding enabled.

    • false with Insights Datadog platform only enabled (no third-party forwarding) .

  5. Test network connectivity: Verify the Insights Agent can reach your third-party observability platform endpoint. Network firewalls or proxies may be blocking outbound connections.

  6. Verify credentials: Ensure the credentials embedded in otel-config.yaml are still valid and have not expired.

  7. Check resource allocation: Verify that you have provided sufficient memory to the container according to your event broker size class. Resource exhaustion can cause the OTel collector to fail.

Network and Firewall Requirements for Third-Party Forwarding

Enabling Insights forwarding of metrics and logs requires you to configure specific network connectivity. If your network uses an HTTP or HTTPS proxy for outbound connections, you can use additional environment variables to configure proxy settings. If you deploy your self-managed event brokers in an air-gapped environment, you should configure additional environment variables to suppress connection attempts.

For more information, see:

Networking Requirements for Insights Forwarding for Self-Managed Event Brokers

When you enable Insights forwarding, you must also ensure the network connectivity listed in the following table is available:

From To Port/Protocol Purpose
Insights Agent Third-party backend

The specific ports and protocols vary by observability backend, for example, OTLP typically uses 4317 for gRPC or 4318 for HTTP. For more information, see your third-party observability platform's documentation.

 

 Telemetry export to your observability platform
Administrator workstation Insights Agent container 8888/TCP (optional) Access to OTel collector self-telemetry metrics for monitoring
Event broker Insights Agent localhost Log and metric collection via SEMP

HTTP/HTTPS Proxy Configuration

If your environment requires an HTTP or HTTPS proxy for outbound connections, you can configure proxy settings using environment variables when deploying the Insights Agent:

  • HTTP_PROXY — HTTP proxy URL (for example, http://proxy.example.com:8080)

  • HTTPS_PROXY — HTTPS proxy URL (for example, https://proxy.example.com:8443)

  • NO_PROXY — Comma-separated list of hosts to bypass the proxy (for example, localhost,127.0.0.1,.local)

See the deployment-specific instructions for how to set these environment variables in your deployment type.

Air-Gapped Deployments

If your software event broker or appliance event broker is deployed in an air-gapped or offline environment using third-party forwarding only, the Insights Agent cannot reach Datadog intake endpoints. This causes repeated error logs as the agent retries /intake/ requests. To suppress these errors, set the following environment variables to false:

Variable Effect
INSIGHTS_AGENT_ENABLE_PAYLOADS_JSON_TO_V1_INTAKE

Stop POSTs to the /intake/ (v1 metadata) endpoint.
INSIGHTS_AGENT_ENABLE_METADATA_COLLECTION

Stop collecting host metadata that feeds the /intake/ endpoint.
INSIGHTS_AGENT_INVENTORIES_ENABLED

Stop agent and integration inventory payloads.
INSIGHTS_AGENT_ENABLE_PAYLOADS_EVENTS

Stop event payloads (also sent via v1 intake).

Setting these variables to false disables the payloads that cause the /intake/ errors. Metrics and logs forwarding to your third-party platform is unaffected.

Only set these variables for third-party forwarding deployments that are air-gapped. If your deployment also pushes to Insights Datadog and you want the host and agent inventory data there, leave these variables enabled.

Environment Variables for Insights Forwarding for Self Managed Brokers

The environment variables listed in the following table control the behavior of the Insights Agent when Insights forwarding is enabled. Most of these variables are included in the configuration package generated by the Cloud Console, but additional variables are available for advanced configurations, including:

Core Forwarding Variables

The variables in the following table specify required properties necessary to enable Insights forwarding for your software or appliance event broker.

Variable Required? Description
INSIGHTS_AGENT_FORWARDING_ENABLED Required Set to true to enable third-party forwarding.
INSIGHTS_AGENT_TELEMETRY_ENABLED Optional

Set to true to enable OTel collector self-telemetry metrics on port 8888 for monitoring and troubleshooting. Default: false.

Dual Forwarding Mode Variables

When using dual forwarding mode (forwarding to both Datadog and a third-party observability platform), set the following environment variables:

Variable Description
INSIGHTS_AGENT_INSIGHTS_FORWARDING_ENABLED

Set to true to enable dual forwarding mode. This variable controls whether the Insights Agent forwards metrics and logs to Datadog additional endpoints.

When set to false, the agent infers logs-off and uses empty additional endpoints (third-party only mode).
INSIGHTS_AGENT_ADDITIONAL_ENDPOINTS

JSON object specifying the Datadog endpoint and API key(s) for metrics.

Format: '{"https://app.<site>":["<api-key>"]}'

For example: '{"https://app.datadoghq.com":["your-api-key"]}'
INSIGHTS_AGENT_LOGS_CONFIG_ADDITIONAL_ENDPOINTS

JSON array specifying the Datadog logs endpoint configuration.

Format: '[{"api_key":"<api-key>","host":"agent-http-intake.logs.<site>","use_compression":true,"compression_level":2}]'

For example: '[{"api_key":"your-api-key","host":"agent-http-intake.logs.datadoghq.com","use_compression":true,"compression_level":2}]'

Proxy and Air-Gapped Variables

See Network and Firewall Requirements for Third-Party Forwarding for information about proxy and air-gapped environment variables.

Monitoring OTel Collector Health

You can monitor the health of your Insights Forwarding setup using the self-telemetry metrics the OTel Collector generates and sends along with your forwarded Insights metrics and logs. You can use the metrics described in the following table to understand whether the OTel collector is running, and receiving and successfully exporting your data. The counters are cumulative, so read them as a rate or delta over time.

See Interpreting OTel Collector Metrics for examples.

Metric What It Reports Expected Value
otelcol.process.uptime

Seconds since the collector process started (monotonic counter; resets to 0 on restart).

Steadily increasing. Reset to ~0 indicates collector restarted. Frequent resets indicate crash-looping. Absent indicates collector is not running.
otelcol.receiver.accepted_metric_points

Metric points successfully accepted into the pipeline.

Rising steadily while the event broker emits metrics. Flat at 0 indicates the collector is not receiving.
otelcol.receiver.refused_metric_points

Metric points refused (could not enter the pipeline, typically due to downstream backpressure or errors).

0 or flat. Rising indicates backpressure or a downstream problem.
otelcol.receiver.accepted_log_records

Log records successfully accepted into the pipeline.

Rising when logs are forwarded. May be 0 for configurations that do not forward logs.
otelcol.receiver.refused_log_records

Log records refused (could not enter the pipeline).

0 or flat.
otelcol.exporter.sent_metric_points

Metric points successfully sent to the destination backend. This is the primary indicator that metrics are reaching your backend.

Rising, tracking accepted. Flat while accepted keeps rising indicates export is broken.
otelcol.exporter.send_failed_metric_points

Metric points that failed to send to the backend.

0 or flat. Rising indicates export failure (connectivity, credentials, or backend rejecting data).
otelcol.exporter.sent_log_records

Log records successfully sent to the backend.

Rising when logs are forwarded. 0 is acceptable if the configuration does not forward logs.
otelcol.exporter.send_failed_log_records

Log records that failed to send to the backend.

0 or flat.

Interpreting OTel Collector Metrics

The following table describes common metric patterns and their likely meanings:

What You See Likely Meaning

No otelcol.* metrics at all, or otelcol.process.uptime absent

The collector is down, or the Insights Agent is not forwarding to your backend at all.

otelcol.process.uptime keeps resetting

The collector is crash-looping. Check container resources and configuration.

otelcol.receiver.accepted_* flat at 0 while otelcol.process.uptime is healthy

The collector is up but not receiving. The Insights Agent is not pushing telemetry into it. Check the Insights Agent and the event broker.

otelcol.receiver.accepted_* rising, but otelcol.exporter.sent_* flat or otelcol.exporter.send_failed_* rising

The collector receives but cannot export. Check network or proxy settings, backend credentials, and backend availability.

otelcol.receiver.refused_* rising

Pipeline backpressure or internal errors. Check container resources and the downstream backend.

Some third-party forwarding configurations do not forward logs. In that case, zero *_log_records values are expected and do not indicate a problem.

Troubleshooting Third-Party Forwarding

Below are some troubleshooting methods you can try if you are having issues with Insights forwarding. If you can't resolve the issues on your own, contact Solace.

Fallback Behavior for Invalid Forwarding Configuration

If you are using dual forwarding mode (forwarding to both your Insights Datadog account and a third-party platform) and the forwarding configuration is invalid or missing, the Insights Agent continues to push metrics and logs to your Insights Datadog account. However, the OTel collector does not forward data to your third-party platform.

Check the Insights Agent logs for errors related to the OTel collector process to diagnose the issue. Common causes include the otel-config.yaml file being missing, unreadable, or containing errors. Verify the file is correctly mounted and contains valid YAML.

Harmless Log Messages

The Insights Agent may emit ERROR or WARN log messages that appear alarming but are expected and harmless. These messages do not affect metric or log collection, or forwarding to third-party platforms. You do not need to act on these messages. The following types of messages can be safely ignored:

  • Remote config protocol test errors

  • Remote config service not initialized or started errors

  • Workloadmeta collectors not ready after retries

  • NTP server unreachable warnings (in air-gapped or network-restricted environments)

Validating Environment Variables Inside Containers

To verify that environment variables are correctly set inside a running container, exec into the Insights Agent container and read the process environment:

Linux containers (Docker/Podman/Appliance):

cat /proc/<process_id>/environ | tr '\0' '\n' | grep INSIGHTS

Where <process_id> is the process ID of the Datadog agent process. This displays all INSIGHTS_AGENT_* environment variables as seen by the running agent process.

Kubernetes:

kubectl exec <pod-name> -- env | grep INSIGHTS

For additional troubleshooting guidance specific to your deployment type, see the validation sections in the deployment-specific documentation.

Upgrading the Insights Agent with Forwarding for Insights Enabled

When upgrading the Insights Agent, you must use the same otel-config.yaml file and environment variables from your original installation. Do not regenerate the configuration unless you are also changing your third-party platform settings.

When a new version of the Insights Agent becomes available, you can upgrade your Insights Agent with forwarding enabled using the same procedure as an upgrade for an Insights Agent without forwarding, using the same otel-config.yaml file and environment variables as when you first deployed the Insights Agent.

Follow the upgrade procedure for your deployment type: