Context Propagation for Distributed Tracing in the PubSub+ JCSMP API

Distributed tracing allows your enterprise applications to trace the flow of messages as they travel from your publisher, through the event mesh and to the receiving application. For a detailed overview, see Distributed Tracing . For information about version requirements, see Distributed Tracing Version Compatibility.

For information about instrumenting your code for distributed tracing, see Instrumenting JCSMP for Distributed Tracing

Instrumenting JCSMP for Distributed Tracing

There are two ways to integrate distributed tracing into an application using the PubSub+ JCSMP API:

• Auto-Instrumentation allows your applications to inject and extract standard OpenTelemetry data without any modifications to your source code.
• Manual Instrumentation involves making changes to your enterprise application's source code. These code modifications allow you to inject and extract additional context, such as Baggage and Trace States, into messages.

Auto-Instrumentation

Auto-instrumentation does not require any application code changes to send telemetry data. You do not need to change the start up parameters in your application or implement any additional libraries.

In order to use auto-instrumentation you need to download the .jar files listed below and place them in your project:

• OpenTelemetry Instrumentation for Java
• PubSub+ OpenTelemetry Integration for Solace JCSMP API

  For OpenTelemetry version compatibility see Distributed Tracing Version Compatibility.

Injecting Context into Outbound Messages

At the command prompt, enter the following command to configure your publishing application to inject OpenTelemetry data into published messages.

java -javaagent:<absolute-path-to-otel-jar>/opentelemetry-javaagent-all-<otel-java-version>.jar \
-Dotel.javaagent.extensions=<absolute-path-to-solace-jar>/solace-opentelemetry-jcsmp-integration-<solace-opentelemetry-version>.jar \
-Dotel.propagators="solace_jcsmp_tracecontext","solace_jcsmp_baggage" \
-Dotel.exporter.otlp.endpoint=<http://localhost:4317> \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.traces.exporter=otlp \
-Dotel.metrics.exporter=none \
-Dotel.resource.attributes="service.name=SolaceJCSMPPublisher" \
-jar <your-jcsmp-publisher-application>.jar 

Where :

• <absolute-path-to-otel-jar>is the absolute file path to the directory containing your OpenTelemetry Java Agent JAR file.
• <absolute-path-to-solace-jar> is the absolute file path to the directory containing your Solace PubSub+ OpenTelemetry JCSMP Integration JAR file.
• solace_jcsmp_tracecontext is the name of the JCSMP custom trace context propagator. This must be explicitly set as “solace_jcsmp_tracecontext” for the OTEL Java Agent to identify the propagator.
• solace_jcsmp_baggage is the name of the JCSMP custom baggage propagator. This must be explicitly set as “solace_jcsmp_baggage” for the OTEL Java Agent to identify the propagator. If you do not want to propagate baggage, omit this property.
• <http://localhost:4317> is the URL of the Collector you are using. The example localhost:4317is a default port for an OpenTelemetry Collector that is running locally. To allow your application to send trace data to an OTLP endpoint you’ll need to update the property otel.exporter.otlp.endpoint=http://localhost:4317 to point to your collector's IP address.
• <your-jcsmp-publisher-application> is the name of the JAR file associated with your JCSMP publisher application.
• <otel-java-version> is the version OpenTelementry Instrumentation Java Library that is compatible with the PubSub+ OpenTelemetry Integration for Solace JCSMP API version(<solace-opentelementry-version> ). For information about the compatibility, see Distributed Tracing Version Compatibility.
• <solace-opentelementry-version> is the PubSub+ OpenTelemetry Integration for Solace JCSMP to use with the OpenTelementry Instrumentation Library (<otel-java-version>). For information about the compatibility, see Distributed Tracing Version Compatibility.

Extracting Context from Received Messages

At the command prompt, enter the following command to configure your queue receiver application to extract OpenTelemetry data from received messages:

java -javaagent:<absolute-path-to-otel-jar>/opentelemetry-javaagent-all-<otel-java-version>.jar \
-Dotel.javaagent.extensions=<absolute-path-to-solace-jar>/solace-opentelemetry-jcsmp-integration-<solace-opentelementry-version>.jar \
-Dotel.propagators="solace_jcsmp_tracecontext","solace_jcsmp_baggage" \
-Dotel.exporter.otlp.endpoint=<http://localhost:4317> \
-Dotel.exporter.otlp.protocol=grpc \
-Dotel.traces.exporter=otlp \
-Dotel.metrics.exporter=none \
-Dotel.resource.attributes="service.name=SolaceJCSMPConsumer" \
-jar <your-jcsmp-consumer-application>.jar

Where :

• <absolute-path-to-otel-jar> is the absolute file path to the directory containing your OpenTelemetry Java Agent JAR file.
• <absolute-path-to-solace-jar> is the absolute file path to the directory containing your PubSub+ OpenTelemetry JCSMP Integration JAR file.
• solace_jcsmp_tracecontext is the name of the JCSMP custom trace context propagator. This must be explicitly set as “solace_jcsmp_tracecontext” for the OTEL Java Agent to identify the propagator.
• solace_jcsmp_baggage is the name of the JCSMP custom baggage propagator. This must be explicitly set as “solace_jcsmp_baggage” for the OTEL Java Agent to identify the propagator. If you do not want to propagate baggage, omit this property.
• <http://localhost:4317> is the URL of the Collector you are using. The example localhost:4317is a default port for an OpenTelemetry Collector that is running locally. To allow your application to send trace data to an OTLP endpoint you’ll need to update the property otel.exporter.otlp.endpoint=http://localhost:4317 to point to your collector's IP address.
• <your-jcsmp-consumer-application> is the name of the JAR file associated with your JCSMP consumer application.
• <otel-java-version> is the version OpenTelementry Instrumentation Java Library that is compatible with the PubSub+ OpenTelemetry Integration for Solace JCSMP API version(<solace-opentelementry-version> ). For information about the compatibility, see Distributed Tracing Version Compatibility.
• <solace-opentelementry-version> is the PubSub+ OpenTelemetry Integration for Solace JCSMP to use with the OpenTelementry Instrumentation Library (<otel-java-version>). For information about the compatibility, see Distributed Tracing Version Compatibility.

When you use auto-instrumentation, the latest version of the opentelemetry-javaagent-all-<otel-java-version>.jar and solace-opentelemetry-jcsmp-integration-<solace-opentelementry-version>.jar should not be added as Maven or Gradle dependencies in your applications. For information about the compatibility, see Distributed Tracing Version Compatibility.

For more information, see:

• Codelab: Getting Started with Solace Distributed Tracing and Context Propagation
• PubSub+ Messaging APIs - Distributed Tracing OpenTelemetry Auto-Instrumentation Span Fields

Manual Instrumentation

Manual Instrumentation involves making changes to your enterprise application's source code, and allows you to inject and extract additional context, such as baggage and trace states, into messages. Context propagation makes it easy to debug and optimize your application. For more information about context propagation in Solace event messages, see Distributed Tracing Context Propagation. The following examples show you how to create spans using the OpenTelemetry API.

How Context Propagation Enables Distributed Tracing in the JCSMP PubSub+ API

In your client application, you can use the OpenTelemetry API to create a span, which contains metadata about an operation in a distributed system. This span is associated with a context, which includes a unique TraceID. Next, when you use a PubSub+ message producer to publish a message, the Solace OTel integration package injects the context, which contains the TraceID, into the message. As the message travels through the event broker and is received by a consuming application, spans are generated at each step and have the same TraceID present in the original message context. When each span is closed in the publishing or consuming application, the Java OpenTelemetry API sends it to an OpenTelemetry collector, which collects, processes and exports the spans to a backend application that correlates the spans using their unique TraceID. A backend application uses the correlated spans to create a trace, which is an end-to-end snapshot detailing how the message traveled through the distributed system. If you do not use context propagation, then backend applications cannot use a unique TraceID to link the spans, making it difficult to trace the flow of messages through the distributed system.

Dependencies

To enable context propagation for distributed tracing, you must first add the Solace PubSub+ OpenTelemetry Integration For Solace JCSMP API library as a dependency in your application. This library automatically adds the OpenTelemetry API and SDK libraries, which are required for context propagation. For OpenTelemetry version compatibility see Distributed Tracing Version Compatibility. Adding the libraries gives you access to the following two interfaces:

• SolaceJCSMPTextMapSetter— This interface allows a TextMapPropagator to inject context into a message.
• SolaceJCSMPTextMapGetter— This interface allows a TextMapPropagator to extract context from a message.

Generating a Send Span on Message Publish

Your publishing application can generate a send span and export it to the OpenTelemetry Collector. The following steps show you how to inject context into a message and generate a send span for a published message:

1. Create a new span and set span attributes with the setAttribute() method. Next, set the current context as the parent of this span. Start the span with the startSpan() method:

final Span sendSpan = tracer
    .spanBuilder("mySolacePublisherApp" + " " + MessagingOperationValues.PROCESS)
    .setSpanKind(SpanKind.CLIENT)
    // Publish to a non-temporary topic endpoint
    .setAttribute(SemanticAttributes.MESSAGING_DESTINATION_KIND, MessagingDestinationKindValues.TOPIC)
    .setAttribute(SemanticAttributes.MESSAGING_TEMP_DESTINATION, false)
    // Set more attributes as needed
    //.setAttribute(...)
    //.setAttribute(...)
    .setParent(Context.current()) // Set the current context as the parent span
    .startSpan();

2. Set the created span from Step 1 (sendSpan in this example) as the new current context. Next, inject the current context into your message and then publish the message. Call the end() method on the span to export the span data:

try (Scope scope = sendSpan.makeCurrent()) {
    final SolaceJCSMPCTextMapSetter setter = new SolaceJCSMPTextMapSetter();
    final TextMapPropagator propagator = openTelemetry.getPropagators().getTextMapPropagator();
    // Inject the current context with send span into the message
    propagator.inject(Context.current(), message, setter);
    // Publish the message to the given topic
    messageProducer.send(message, messageDestination);
} catch (Exception e) {
    sendSpan.recordException(e);                          // Span can record an exception 
    sendSpan.setStatus(StatusCode.ERROR, e.getMessage()); // Set span status as ERROR/FAILED
} finally {
    sendSpan.end(); // The span data is exported after you call end()
}

Generating a Receive Span on Message Receive

Your consuming application can generate a receive span and then export it to the OpenTelemetry Collector. The following steps show you how to extract tracing context from a received message and generate a receive span:

1. Use a SolaceJCSMPCTextMapGetter to extract any existing context from the received message:

final SolaceJCSMPTextMapGetter getter = new SolaceJCSMPTextMapGetter();
final Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
       .extract(Context.current(), message, getter);

2. Set the extracted context as the current context with the makeCurrent() method. Next create a child span (receiveSpan in this example) and set the extracted context as the parent of that child span. Start the span using the startSpan() method:

try (Scope scope = extractedContext.makeCurrent()) {
    // Create a child span and set extracted/current context as the parent of this span
    final Span receiveSpan = tracer
        .spanBuilder("mySolaceReceiverApp" + " " + MessagingOperationValues.RECEIVE)
        .setSpanKind(SpanKind.CLIENT)
        // For the case the message was received on a non-temporary queue endpoint
        .setAttribute(SemanticAttributes.MESSAGING_DESTINATION_KIND,
            MessagingDestinationKindValues.QUEUE)
        .setAttribute(SemanticAttributes.MESSAGING_TEMP_DESTINATION, false)
        // Set more attributes as needed
        //.setAttribute(...)
        //.setAttribute(...)
        // Creates a parent-child relationship to a message publisher's application span
        .setParent(extractedContext)
        // Starts span
        .startSpan();

    // Try-catch continues in the next step...
}

3. Accept and process the received message, then call the end() method on the receive span to export the span data:

try {
    // Do something with the message in a callback function
    messageProcessor.accept(receivedMessage);
} catch (Exception e) {
    receiveSpan.recordException(e);                          // Span can record an exception
    receiveSpan.setStatus(StatusCode.ERROR, e.getMessage()); // and set span status as ERROR
} finally {
    receiveSpan.end(); // Span data is exported when you call span.end()
}