Manual Configuration for Insights Agent for Self-Managed Event Brokers

Obtaining the Insights Agent Image

You can obtain the Insights Agent image using one of the following methods:

• Solace Container Registry Solace recommends pulling the image directly from the Solace Container Registry (gcr.io). This method requires registry credentials. For more information, see Using the Solace Container Registry.

• products.solace.com — Download the image archive from products.solace.com and manually load it into your container runtime. For more information, see Downloading from products.solace.com.

Using the Solace Container Registry

To pull the Insights Agent image from the Solace Container Registry (gcr.io), you need registry credentials. If you have access to the Cloud Console, you can download the credentials file. If you cannot access Cloud Console, contact Solace to obtain credentials. For more information, see Connectivity Model for Kubernetes Deployments.

cat /path/to/<datacenter-id>-registry-credentials.json | podman login -u _json_key --password-stdin gcr.io

Insights Agent Environment Variables

When you cannot use the Cloud Console to generate the Insights Agent properties file, you must manually configure the environment variables. This section provides a consolidated reference of all environment variables.

For standard Insights deployments (no third-party forwarding), you only need to configure the Core Connection Variables. The Insights Forwarding Variables and configuration files (otel-config.yaml, logs.yaml) are only required when enabling third-party forwarding.

Core Connection Variables

The following variables are required for all Insights Agent deployments:

Insights Forwarding Variables

The following variables are required only when enabling third-party forwarding. If you are deploying the Insights Agent in a standard configuration without forwarding (Datadog only), skip this section.

Optional Variables

The following variables are optional for advanced configurations:

Proxy Configuration

If your environment uses an HTTP or HTTPS proxy for outbound connections, you must configure proxy settings so the Insights Agent can reach your observability platform. Without proper proxy configuration, the Insights Agent cannot forward metrics and logs.

Configure the following environment variables when deploying the Insights Agent in an environment with a proxy:

Example Docker command with proxy configuration:

docker run \
  -e INSIGHTS_AGENT_HTTP_PROXY="http://proxy.example.com:8080" \
  -e INSIGHTS_AGENT_HTTPS_PROXY="https://proxy.example.com:8443" \
  -e INSIGHTS_AGENT_NO_PROXY="localhost,127.0.0.1,broker-hostname" \
  # --- other environment variables ---
  gcr.io/gcp-maas-prod/solace-insights-agent:<version>

Air-Gapped Environment Variables

Set the variables listed in this section for air-gapped third-party-only deployments where the agent cannot reach Datadog. You should not set them for dual forwarding mode deployments.

If you deploy in an air-gapped environment where the Insights Agent cannot reach Datadog endpoints, configure the following variables to suppress connection error messages:

For more information about network requirements and air-gapped deployments, see Network and Firewall Requirements for Third-Party Forwarding.

Supported Exporters

The exporters: block is the only backend-specific part of otel-config.yaml. This section provides ready-to-use exporter blocks for supported backends. See the section for your exporter:

• OTLP/HTTP (Generic)

• OTLP/gRPC

• Dynatrace

• New Relic

• Elasticsearch

• Splunk HEC

• Prometheus (Remote Write) — Metrics Only

An example is also provided. For more information, see Example: Dynatrace Configuration.

OTLP/HTTP (Generic)

The exporter name (for example, otlp_http/metrics) must match what you list under the pipeline's exporters:. Names with a / are type/identifier - keep the type prefix.

Copy

otlp_http/metrics:
  endpoint: "https://<your-otlp-endpoint>:4318"
  headers:                       # optional
    Authorization: "Bearer <token>"
  sending_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # optional
    insecure: false
    insecure_skip_verify: true

OTLP/gRPC

Copy

otlp_grpc/metrics:
  endpoint: "https://<your-otlp-endpoint>:4317"
  headers:                       # optional
    Authorization: "Bearer <token>"
  sending_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # optional
    insecure: false
    insecure_skip_verify: true

Dynatrace

Copy

otlp_http/metrics:
  endpoint: "https://<env-id>.live.dynatrace.com/api/v2/otlp"
  headers:
    Authorization: "Api-Token <api_token>"
  sending_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # optional
    insecure: false
    insecure_skip_verify: true

New Relic

Copy

otlp_http/metrics:
  endpoint: "https://otlp.nr-data.net"
  headers:
    api-key: "<api_key>"
  compression: zstd
  sending_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # required for New Relic
    insecure: false
    insecure_skip_verify: true

Elasticsearch

Copy

otlp_http/metrics:
  endpoint: "https://<elasticsearch-otlp-endpoint>"
  headers:
    Authorization: "ApiKey <api_key>"
  sending_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # optional
    insecure: false
    insecure_skip_verify: true

Splunk HEC

Copy

splunk_hec/metrics:
  token: "<hec_token>"
  endpoint: "https://<splunk-host>:8088/services/collector"
  source: "solace-insights"
  use_multi_metric_format: true
  sourcetype: "<sourcetype>"     # optional
  index: "<index>"               # optional
  disable_compression: false     # optional
  timeout: "10s"                 # optional
  sending_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # optional
    insecure: false
    insecure_skip_verify: true

Prometheus (Remote Write) — Metrics Only

Prometheus is metrics only (no logs pipeline) and uses CUMULATIVE temporality (deltatocumulative). See Metrics-Only Configuration.

Copy

prometheusremotewrite/metrics:
  endpoint: "https://<prometheus-host>/api/v1/write"
  headers:                       # only when basic_auth is set
    Authorization: "Basic <base64(user:password)>"
  remote_write_queue:
    enabled: true
  retry_on_failure:
    enabled: true
  tls:                           # optional
    insecure: false
    insecure_skip_verify: true

Example: Dynatrace Configuration

The following is a complete working example of an otel-config.yaml file that exports both metrics and logs to Dynatrace:

exporters:
  otlp_http/metrics:
    headers:
      Authorization: "Api-Token <ACCESS_TOKEN>"
    endpoint: "https://<env-id>.live.dynatrace.com/api/v2/otlp"
    tls:
      insecure: false
      insecure_skip_verify: true
    sending_queue:
      enabled: true
    retry_on_failure:
      enabled: true
  otlp_http/logs:
    headers:
      Authorization: "Api-Token <ACCESS_TOKEN>"
    endpoint: "https://<env-id>.live.dynatrace.com/api/v2/otlp"
    tls:
      insecure: false
      insecure_skip_verify: true
    sending_queue:
      enabled: true
    retry_on_failure:
      enabled: true

# --- (receivers and processors sections from template) ---

service:
  extensions:
  - file_storage/filelog
  - health_check
  # --- (telemetry section from template) ---
  pipelines:
    metrics:
      receivers:
      - datadog
      processors:
      - memory_limiter
      - resource/metrics
      - attributes/metrics
      - transform/cleanup
      - filter/otel
      - transform/fix_timestamps
      - cumulativetodelta
      - batch
      exporters:
      - otlp_http/metrics
    logs:
      receivers:
      - file_log/otlcol
      - file_log/debug
      - file_log/system
      - file_log/command
      processors:
      - memory_limiter
      - attributes/metrics
      - batch
      exporters:
      - otlp_http/logs

Choosing the Metric Temporality Processor

The Insights Agent produces cumulative metrics. The OTel collector must convert these to the temporality (how each data point's value relates to time) that your destination backend expects. Use the following table to determine which processor to use:

If you are unsure, use cumulativetodelta. DELTA is the default for all backends except Prometheus. The processor name defined under processors: must match the name listed in the metrics pipeline.

If you need to use deltatocumulative for a cumulative-native backend (Prometheus), replace the cumulativetodelta processor block with:

deltatocumulative:
  max_stale: 30m

Then update the metrics pipeline to reference deltatocumulative instead of cumulativetodelta.

Metrics-Only Configuration

You do not need a logs.yaml file when forwarding metrics only.

If you want to forward only metrics (no logs) to your third-party backend, make the following changes to the template:

1. In the exporters: section, define only your metrics exporter; do not enter a logs exporter.

2. Delete the entire logs: section pipeline under service.pipelines (you must keep metrics:).

3. Delete every file_log/* receiver (file_log/otlcol, file_log/debug, file_log/event, file_log/agent, file_log/system, file_log/command). The OTel collector fails to start if a receiver is configured but not used by any pipeline.

4. Delete the file_storage/filelog extension from both extensions: and service.extensions: sections. Keep health_check.

The structural parts should look like the following example:

receivers:
  datadog:
    endpoint: localhost:6000
    read_timeout: 60s
    write_timeout: 60s
    idle_series_timeout: 15m
    idle_series_cleanup_interval: 5m
    intake:
      behavior: proxy
      proxy:
        api:
          key: "<DATADOG_API_KEY>"
          site: "datadoghq.com"
          fail_on_invalid_key: false
  # (all file_log/* receivers deleted)

processors:
  # >>> FILL IN [temporality] -------------------------------------------------
  # Choose ONE metric-temporality processor based on the backend you export
  # metrics to. See "Choosing the Metric Temporality Processor" section.
  #   * DELTA backends (Dynatrace, New Relic, Splunk, Elasticsearch, Datadog,
  #     generic OTLP) - KEEP cumulativetodelta below.
  #   * CUMULATIVE backends (Prometheus remote-write) - DELETE cumulativetodelta
  #     and use deltatocumulative instead. Then change cumulativetodelta to
  #     deltatocumulative in the metrics pipeline at the bottom.
  cumulativetodelta:
     max_staleness: 30m
  # <<< END FILL IN [temporality] ---------------------------------------------

  memory_limiter:
    check_interval: 0.75s
    # FILL IN [memory]: set limit_mib to 80% and spike_limit_mib to 20% of the
    # OTel collector container's memory limit, in MiB.
    # Example for a 512Mi limit: limit_mib=410, spike_limit_mib=102
    limit_mib:           # FILL IN
    spike_limit_mib:     # FILL IN

  resource/metrics:
    attributes:
     - action: upsert
       key: service.name
       value: maas-vmr
     - action: upsert
       key: host.name
       value: ${DD_HOSTNAME}
 
  transform/cleanup:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "check_name") where attributes["check_name"] == ""

  transform/fix_timestamps:
    metric_statements:
      - context: datapoint
        statements:
         - set(time_unix_nano, UnixNano(Now())) where time_unix_nano == 0

  attributes/metrics:
    actions:
      - action: insert
        key: third_party_forwarding_enabled
        value: true
    # >>> FILL IN [host-tags] ---------------------------------------------------
    # Add one "insert" action per host tag you want on every metric and log.
    # Replace the examples below with your own tags.
    # - action: insert
    #   key: your_tag_name
    #   value: "your_tag_value"
    # <<< END FILL IN [host-tags] -----------------------------------------------

  filter/otel:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - .*system\.post-status$
          - .*system\.cpu.*
          - .*system\.disk.*
          - .*system\.core.*
          - .*system\.io.*
          - .*system\.fs.*
          - .*system\.mem.*
          - .*system\.paging.*
          - .*system\.processes.*
          - .*system\.load.*
          - .*system\.uptime.*
          - .*system\.swap.*
          - .*system\.net.*
          - .*trace_agent.stats_.*
          - .*_scrape_.*
          - http\.client\.request\.body\.size.*
          - http\.client\.request\.duration.*
          - http\.server\.request\.body\.size.*
          - http\.server\.request\.duration.*
          - http\.server\.response\.body\.size.*

  batch:
    send_batch_max_size: 512
    send_batch_size: 256
    timeout: 5s

extensions:
  # file_storage/filelog deleted
  health_check:
    endpoint: "0.0.0.0:13133"
    path: "/health"

service:
  extensions:
  - health_check
  pipelines:
    metrics:
      receivers: [datadog]
      processors: [memory_limiter, resource/metrics, attributes/metrics, transform/cleanup, filter/otel, transform/fix_timestamps, cumulativetodelta, batch]
      exporters: [otlp_http/metrics]   # your metrics exporter name
  # (logs pipeline deleted)

Datadog Agent Logs Configuration Template for Insights Dual Forwarding Mode

The logs.yaml file is required for dual forwarding mode only (forwarding to both Insights and a third-party platform). If you are deploying standard Insights or using third-party only mode, you do not need this file.

Mount this file at /etc/datadog-agent/conf.d/solace.d/logs.yml on the Insights Agent container.

# =============================================================================
# logs.yaml - Datadog Agent log tailing for Insights Agent
#             with THIRD-PARTY FORWARDING (Dual Forwarding Mode)
# -----------------------------------------------------------------------------
# Static config. No secrets to fill in here.
# Mount this file at /etc/datadog-agent/conf.d/solace.d/logs.yml
# =============================================================================
logs:
  - path: /jail/logs/system.log
    service: maas-vmr
    source: system.log
    type: file
  - path: /jail/logs/command.log
    service: maas-vmr
    source: command.log
    type: file
  - path: /jail/logs/debug.log
    service: maas-vmr
    source: debug.log
    type: file
    log_processing_rules:
    - type: include_at_match
      name: include_limited_eventLogs
      pattern: "<local0.alert>\
|<local0.crit>\
|<local0.err>"
  - path: /etc/datadog-agent/output-logs.log
    service: otelcol-agent
    source: otelcol-output-logs.log
    type: file
    start_position: beginning
  - path: /etc/datadog-agent/error-output-logs.log
    service: otelcol-agent
    source: otelcol-error-output-logs.log
    type: file
    start_position: beginning
  - path: /var/log/datadog/agent.log
    service: solace-datadog-agent
    source: agent.log
    type: file
  - path: /jail/logs/event.log
    service: maas-vmr
    source: event.log
    type: file
    log_processing_rules:
    - type: include_at_match
      name: include_limited_eventLogs
      pattern: "SYSTEM_AUTHENTICATION_SESSION_DENIED\
|SYSTEM_CLIENT_ACL_CONNECT_DENIAL\
|SYSTEM_CLIENT_ACL_PUBLISH_DENIAL\
|SYSTEM_CLIENT_ACL_SUBSCRIBE_DENIAL\
|SYSTEM_CLIENT_CONNECT_AUTH_FAIL\
|SYSTEM_CLIENT_CONNECT_FAIL\
|VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_EXCEED\
|VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_HIGH\
|VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_HIGH_CLEAR\
|VPN_AD_PARTITIONED_QUEUE_REBALANCE_STARTED\
|VPN_AD_PARTITIONED_QUEUE_REBALANCE_COMPLETED\
|VPN_AD_REPLAY_STATE_TRANSITION\
|VPN_AD_REPLAY_STATE_TRANSITION_TO_FAILED\
|VPN_BRIDGING_LINK_TTL_EXCEEDED\
|VPN_BRIDGING_LINK_TTL_EXCEEDED_CLEAR\
|VPN_CLIENT_USERNAME_CONNECT_FAIL\
|VPN_RDP_RC_CONN_DOWN\
|VPN_RDP_RC_CONN_UP\
|VPN_RDP_RC_DOWN\
|VPN_RDP_RC_UP\
|VPN_RDP_RDP_DOWN\
|VPN_RDP_RDP_UP\
|CLIENT_AD_EGRESS_FLOWS_HIGH\
|CLIENT_AD_EGRESS_FLOWS_HIGH_CLEAR\
|CLIENT_AD_INGRESS_FLOWS_HIGH\
|CLIENT_AD_INGRESS_FLOWS_HIGH_CLEAR\
|CLIENT_AD_MAX_EGRESS_FLOWS_EXCEEDED\
|CLIENT_AD_MAX_INGRESS_FLOWS_EXCEEDED\
|CLIENT_AD_TRANSACTED_SESSIONS_EXCEED\
|CLIENT_AD_TRANSACTED_SESSIONS_HIGH\
|CLIENT_AD_TRANSACTED_SESSIONS_HIGH_CLEAR\
|CLIENT_AD_TRANSACTED_SESSION_FAIL\
|CLIENT_AD_TRANSACTIONS_EXCEED\
|CLIENT_AD_TRANSACTIONS_HIGH\
|CLIENT_AD_TRANSACTIONS_HIGH_CLEAR\
|CLIENT_CLIENT_ACK_NOT_ALLOWED\
|CLIENT_CLIENT_BIND_FAILED\
|CLIENT_CLIENT_CREATE_ENDPOINT_FAILED\
|CLIENT_CLIENT_DISCONNECT:.*#bridge\
|CLIENT_CLIENT_EGRESS_MSG_DISCARD\
|CLIENT_CLIENT_LARGE_MESSAGE\
|CLIENT_CLIENT_MESSAGE_TOO_BIG\
|CLIENT_CLIENT_NAME_CHANGE_FAILED\
|CLIENT_CLIENT_PARSE_ERROR\
|CLIENT_CLIENT_SUBSCRIPTIONS_HIGH"