Sending Application Trace Spans to Insights Using a Self-Managed OpenTelemetry Collector

You can use a self-managed OpenTelemetry (OTEL) collector to send application trace spans to your Solace Insights Datadog account. Sending application trace spans to your Insights Datadog account allows you to correlate application trace spans with event broker service trace spans in a unified view in Datadog.

By deploying your own OpenTelemetry collector with credentials scoped to your Insights Datadog account, you gain full distributed tracing across your event-driven architecture without requiring direct connectivity to the infrastructure hosting your event broker services.

Using a self-managed OpenTelemetry collector to send application trace spans to your Insights Datadog account provides the following benefits:

Unified Observability: You can view application and event broker service telemetry together in your Insights Datadog account.
Self-Serve Configuration: Using the Insights Advanced Manager role, you can use your own Datadog API credentials to configure sending trace spans without needing to contact Solace.
Enhanced Troubleshooting: Tracing message flows from publishers through event broker service to consumers provides enhanced context when troubleshooting messaging issues.
Flexible Deployment: You can deploy and maintain your OpenTelemetry collector in infrastructure you already own and are comfortable managing, be that a Kubernetes cluster, virtual machine, or container environment.

To understand the flow of sending application trace spans to Insights, see Architectural Overview of Sending Traces to Insights.

Before configuring your self-managed OpenTelemetry collector to send application trace spans to your Insights Datadog account, review the Considerations for Sending Application Trace Spans to Insights.

Sending application trace spans consumes part of your Insights trace span quota. For more information, see Trace Span Usage Limits.

Sending application trace spans to Insights requires:

configuring sending of application traces and spans to Insights.
validating application trace and span delivery to Insights.

For troubleshooting and operational guidance related to configuring and sending application trace spans to Insights, see Troubleshooting and Operational Guidance.

Architectural Overview of Sending Traces to Insights

After you have configured your self-managed OpenTelemetry collector to send application traces to your Insights Datadog account, the application tracing flow operates according to the following steps:

Your instrumented applications emit trace spans using the OpenTelemetry Protocol (OTLP).
Your self-managed OpenTelemetry collector receives trace spans via gRPC or HTTP.
The OpenTelemetry collector exports the trace spans to Datadog using your Insights-scoped API credentials.
The trace spans appear in your Insights Datadog account alongside existing event broker service trace spans.
Proper naming and tagging conventions enable cross-component trace span assembly within Datadog.

Using consistent event broker service names and including message-bus context tags (for example, broker host, VPN, topic, etc.) enables your application trace spans to join with your event broker service trace spans in the same distributed trace.

Considerations for Sending Application Trace Spans to Insights

Be aware of the following considerations when using a self-managed OpenTelemetry collector to send application trace spans to Insights:

You must have an Insights subscription to send application trace spans to Insights. To get Insights contact Solace.
You must be using distributed tracing for Solace Cloud to achieve a full distributed trace with event broker service trace spans.
You must have the Insights Advanced Manager role. To get the Insights Advanced Manager role, contact Solace.
You are responsible for deploying, hosting, and managing the OpenTelemetry collector used to collect and send application trace spans within your own infrastructure.
Your self-managed OpenTelemetry collector must have network egress access to Datadog intake endpoints using HTTPS on port 443.
You must be familiar with OpenTelemetry configuration and deployment patterns. You can read more about installing and configuring the OpenTelemtery collector to send telemetry data to Datadog in the Datadog documentation.
Your Insights account comes with a default limit of indexed and ingested trace spans you can use. For more information, see Trace Span Usage Limits.

Trace Span Usage Limits

The trace span data you send to your Insights Datadog account consumes ingested and indexed span quotas. Your Insights account comes with indexed and ingested span limits outlined in the following table:

Quota Type	Allocation Per Month	Retention Period
Indexed Trace Spans	1 million spans per month	15 days
Ingested Trace Spans	150 GB per month	N/A

Indexed and ingested trace span usage beyond the limits in the preceding table requires a Solace Insights Metrics Expansion package. The expansion package includes 5,000 additional custom metrics and increases your limits as outlined in the following table:

Quota Type	Allocation Per Month	Retention Period
Indexed Trace Spans	10 million spans	15 days
Ingested Trace Spans	250 GB	N/A

To get the Solace Insights Metric Expansion, contact Solace.

Controlling Limit Usage

To manage limit usage and stay within your quota limits, Solace suggest doing the following:

Apply tail-based sampling to reduce the volume of low-value trace spans.
Reduce high-cardinality attributes (such as unique identifiers) that increase indexing costs.
Monitor trace span ingestion and indexed trace span usage in your Insights Datadog account to avoid overage charges.

Configuring Sending of Application Trace Spans to Insights

Log into your Insights Datadog account using the Insights Advanced Manager role.
Navigate to Personal Settings > Organization Settings > API Keys.
Click +New Key.
In the New API Key dialog, enter a descriptive name for the key and click Create Key.
Copy the API key and store it securely. Restrict its usage to the OpenTelemetry exporter. Solace recommends rotating your API key regularly.
Deploy your OpenTelemetry collector in your chosen infrastructure: Kubernetes, virtual machine, or container environment. Configure the processors (batch) and the Datadog exporter with your Insights Datadog account credentials. The following example shows a basic OpenTelemetry collector configuration for forwarding traces to Insights, where:
- ${DD_SITE} is the Datadog regional site for your Insights account.
- ${DD_API_KEY} is the Datadog API key you generated earlier.
```
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true
  debug:
    verbosity: detailed
  datadog:
    api:
      site: ${DD_SITE}
      key: ${DD_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, debug, datadog]
```

Validating Application Trace Delivery to Insights

After deploying your OpenTelemetry collector and assuming your applications are instrumented, you can validate that trace spans are flowing to your Insights Datadog account with the following steps:

Log into your Insights Datadog account using the Insights Advanced Manager role.
Navigate to APM > Traces.
Use the Search for filters to locate traces by service name (for example, MyFirstService).
Open a trace and verify that both application and event broker service trace spans appear in the same distributed trace.
Confirm that message-bus context tags are present on application spans.

If traces do not appear:

Check the OpenTelemetry collector logs for export errors.
Verify that you are using the correct Datadog API key and are sending traces to the correct Datadog regional site.
Ensure network connectivity from the OpenTelemetry collector to Datadog intake endpoints is configured correctly.

Troubleshooting and Operational Guidance

This section outlines some troubleshooting and operational guidance recommended by Solace, including:

Security and Compliance Considerations
Scaling the OpenTelemetry Collector
Resilience and Reliability

Security and Compliance Considerations

When sending application trace spans to Insights, Solace recommends following these security best practices:

Credential Management: Store your Datadog API key in a secret manager. Rotate API keys regularly and restrict access to the OpenTelemetry collector.
PII Handling: Scrub sensitive fields (such as personally identifiable information) at the SDK instrumentation level or using the OpenTelemetry collector attributes processor before export.
Network Controls: Allowlist Datadog endpoints, enforce TLS 1.2 or higher, and apply mutual TLS if required by your security policy.

Scaling the OpenTelemetry Collector

Consider monitoring collector performance metrics when sending application trace spans to Insights to ensure adequate capacity:

Queue Length: High queue depth indicates insufficient throughput.
Dropped Spans: Dropped trace spans are a signal that the OpenTelemetry collector cannot keep up with trace span volume.
Exporter Latency: Elevated latency may indicate network or Datadog API issues.

If you're encountering any of these issues, consider deploying separate collectors per environment (development, staging, production) to isolate trace span data and improve fault tolerance.

Resilience and Reliability

Consider the following options to increase resilience and reliability when sending application trace spans to Insights using a self-managed OpenTelemetry collector:

Enable batching and retries in the Datadog exporter to handle transient failures.
Use persistent queues (where supported) to prevent trace loss during collector restarts.

Provide feedback