Context Propagation for Distributed Tracing

Distributed tracing allows your enterprise applications to trace the flow of messages as they travel from your publisher, through the event mesh and to the receiving application. For a detailed overview see Distributed Tracing . For information about version requirements, see Distributed Tracing Version Compatibility.

For information about instrumenting your code for distributed tracing, see:

• Instrumenting JCSMP for Distributed Tracing
• Instrumenting .NET for Distributed Tracing
• Instrumenting JavaScript and Node.js for Distributed Tracing

Instrumenting JCSMP for Distributed Tracing

Manual Instrumentation involves making changes to your enterprise application's source code, and allows you to inject and extract additional context, such as baggage and trace states, into messages. Context propagation makes it easy to debug and optimize your application. For more information about context propagation in Solace event messages, see Distributed Tracing Context Propagation. The following examples show you how to create spans using the OpenTelemetry API.

How Context Propagation Enables Distributed Tracing in the JCSMP PubSub+ API

In your client application, you can use the OpenTelemetry API to create a span, which contains metadata about an operation in a distributed system. This span is associated with a context, which includes a unique TraceID. Next, when you use a PubSub+ message producer to publish a message, the Solace OTel integration package injects the context, which contains the TraceID, into the message. As the message travels through the event broker and is received by a consuming application, spans are generated at each step and have the same TraceID present in the original message context. When each span is closed in the publishing or consuming application, the Java OpenTelemetry API sends it to an OpenTelemetry collector, which collects, processes and exports the spans to a backend application that correlates the spans using their unique TraceID. A backend application uses the correlated spans to create a trace, which is an end-to-end snapshot detailing how the message traveled through the distributed system. If you do not use context propagation, then backend applications cannot use a unique TraceID to link the spans, making it difficult to trace the flow of messages through the distributed system.

Dependencies

To enable context propagation for distributed tracing, you must first add the Solace PubSub+ OpenTelemetry Integration For Solace JCSMP API library as a dependency in your application. This library automatically adds the OpenTelemetry API and SDK libraries, which are required for context propagation. For OpenTelemetry version compatibility see Distributed Tracing Version Compatibility. Adding the libraries gives you access to the following two interfaces:

• SolaceJCSMPTextMapSetter— This interface allows a TextMapPropagator to inject context into a message.
• SolaceJCSMPTextMapGetter— This interface allows a TextMapPropagator to extract context from a message.

Generating a Send Span on Message Publish

Your publishing application can generate a send span and export it to the OpenTelemetry Collector. The following steps show you how to inject context into a message and generate a send span for a published message:

1. Create a new span and set span attributes with the setAttribute() method. Next, set the current context as the parent of this span. Start the span with the startSpan() method:

final Span sendSpan = tracer
	.spanBuilder("mySolacePublisherApp" + " " + MessagingOperationValues.PROCESS)
	.setSpanKind(SpanKind.CLIENT)
	//Publish to a non-temporary topic endpoint
	.setAttribute(SemanticAttributes.MESSAGING_DESTINATION_KIND, MessagingDestinationKindValues.TOPIC)
	.setAttribute(SemanticAttributes.MESSAGING_TEMP_DESTINATION, false)
	//Set more attributes as needed
	//.setAttribute(...)
	//.setAttribute(...)
	.setParent(Context.current()) //Set the current context as the parent span
	.startSpan();

2. Set the created span from Step 1 (sendSpan in this example) as the new current context. Next, inject the current context into your message and then publish the message. Call the end() method on the span to export the span data:

try (Scope scope = sendSpan.makeCurrent()) {
	final SolaceJCSMPCTextMapSetter setter = new SolaceJCSMPTextMapSetter();
	final TextMapPropagator propagator = openTelemetry.getPropagators().getTextMapPropagator();
	//Inject the current context with send span into the message
	propagator.inject(Context.current(), message, setter);
	//Publish the message to the given topic
	messageProducer.send(message, messageDestination);
} catch (Exception e) {
	sendSpan.recordException(e);                          //Span can record an exception 
	sendSpan.setStatus(StatusCode.ERROR, e.getMessage()); //Set span status as ERROR/FAILED
} finally {
	sendSpan.end(); //The span data is exported after you call end()
}	

Generating a Receive Span on Message Receive

Your consuming application can generate a receive span and then export it to the OpenTelemetry Collector. The following steps show you how to extract tracing context from a received message and generate a receive span:

1. Use a SolaceJCSMPCTextMapGetter to extract any existing context from the received message:

final SolaceJCSMPTextMapGetter getter = new SolaceJCSMPTextMapGetter();
final Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
	.extract(Context.current(), message, getter);

2. Set the extracted context as the current context with the makeCurrent() method. Next create a child span (receiveSpan in this example) and set the extracted context as the parent of that child span. Start the span using the startSpan() method:

try (Scope scope = extractedContext.makeCurrent()) {
	//Create a child span and set extracted/current context as parent of this span
	final Span receiveSpan = tracer
		.spanBuilder("mySolaceReceiverApp" + " " + MessagingOperationValues.RECEIVE)
		.setSpanKind(SpanKind.CLIENT)
		// for the case the message was received on a non temporary queue endpoint
		.setAttribute(SemanticAttributes.MESSAGING_DESTINATION_KIND,
		MessagingDestinationKindValues.QUEUE)
		.setAttribute(SemanticAttributes.MESSAGING_TEMP_DESTINATION, false)
		//Set more attributes as needed
		//.setAttribute(...)
		//.setAttribute(...)
		// creates a parent child relationship to a message publisher's application span is any
		.setParent(extractedContext)
		// starts span
		.startSpan();
//Try catch continues in next step...

3. Accept and process the received message, then call the end() method on the receive span to export the span data:

	try {
		//Do something with the message in a callback function
		messageProcessor.accept(receivedMessage);
	} catch (Exception e) {
		receiveSpan.recordException(e);                          //Span can record an exception 
		receiveSpan.setStatus(StatusCode.ERROR, e.getMessage()); //and set span status as ERROR
	} finally {
		receiveSpan.end(); //Span data is exported when you call span.end()
	}
}

Instrumenting .NET for Distributed Tracing

Manual Instrumentation involves making changes to your enterprise application's source code, and allows you to inject and extract additional context, such as baggage and trace states, into messages. Context propagation makes it easy to debug and optimize your application. For more information about context propagation in Solace event messages, see Distributed Tracing Context Propagation. The following examples show you how to create spans using the OpenTelemetry API.

How Context Propagation Enables Distributed Tracing in the .NET PubSub+ API

In your client application, you can use the OpenTelemetry API to create a span, which contains metadata about an operation in a distributed system. This span is associated with a context, which includes a unique TraceID. Next, when you use a PubSub+ message producer to publish a message, the Solace OTel integration package injects the context, which contains the TraceID, into the message. As the message travels through the event broker and is received by a consuming application, spans are generated at each step and have the same TraceID present in the original message context. When each span is closed in the publishing or consuming application, the .NET OpenTelemetry API sends it to an OpenTelemetry collector, which collects, processes and exports the spans to a backend application that correlates the spans using their unique TraceID. A backend application uses the correlated spans to create a trace, which is an end-to-end snapshot detailing how the message traveled through the distributed system. If you do not use context propagation, then backend applications cannot use a unique TraceID to link the spans, making it difficult to trace the flow of messages through the distributed system.

Dependencies

To enable context propagation for distributed tracing, you must first add the Solace .NET OpenTelemetry Integration NuGet package to your application. For OpenTelemetry version compatibility see Distributed Tracing Version Compatibility. The Solace .NET OpenTelemetry Integration NuGet package contains the SolaceMessageCarrier class that gives your application access to the following:

• SolaceMessageCarrier.Setter
  • use with the OpenTelemetry API's TraceContextPropagator to inject context into an IMessage.
  • use with the OpenTelemetry API's BaggagePropagatorto inject baggage into an IMessage.
• SolaceMessageCarrier.Getter
  • use with the OpenTelemetry API's TraceContextPropagator to extract context from an IMessage.
  • use with the OpenTelemetry API's BaggagePropagatorto extract baggage from an IMessage.

To use context propagation in the .NET PubSub+ API, include the following packages in your application:

using OpenTelemetry.Context.Propagation;   // Required for trace propagation
using OpenTelemetry;			   // Contains struct required for injecting and extracting Baggage 
using Solace.Messaging.Trace.Propagation;  // Required for using context propagation in Solace APIs
using SolaceSystems.Solclient.Messaging;   // Required for using the .NET PubSub+ API
using System.Diagnostics;		   // Required for creating activity instances 

System.Diagnostics.Activity in the .NET PubSub+ API

Microsoft's System.Diagnostics.Activity class is used for distributed tracing in the PubSub+ .NET API. An instance of this class, an activity, is similar to a span and represents an operation inside your application, for example publishing or receiving a message. These activities allow backend applications, like Jaeger or Datadog, to trace the flow of events across distributed systems. To create an activity, you must call the StartActivity(String, ActivityKind)method on an ActivitySource object:

ActivitySource activitySource;
using var activity = activitySource.StartActivity("activityName", ActivityKind.Producer);	

You can also set optional attributes on an activity with the following methods to provide additional details or custom information about an activity's execution:

• SetTag(String, Object)—Set custom key-value attributes on an activity to provide additional information.

  activity?.SetTag("myKey1", "myValue1");				

• AddEvent(ActivityEvent)—Add an event to an activity's time line. This method takes an ActivityEvent instance as a parameter, which represents an event that occurred during the activity's execution. The example below shows an exception event being created and added to an activity:

  ActivityTagsCollection tagsCollection = new()
  {
  	{ "exception.message", exceptionMessage }
  };
  activity?.AddEvent(new ActivityEvent("exceptions", tags: tagsCollection)); 

• SetStatus(ActivityStatusCode, String)—Set the status of an activity. Takes an ActivityStatusCode instance as a parameter, which represents the current status of the activity:

  activity?.SetStatus(ActivityStatusCode.Error, "statusErrorMessage");  	// Status code indicating an error is encountered during the operation.
  activity?.SetStatus(ActivityStatusCode.OK, "statusOKMessage"); 	// Status code indicating the operation has been validated and completed successfully.			

For more information see System.Diagnostics.Activity on Microsoft's .NET documentation page

Inject Context into an Outbound Message

The following steps show you how to inject context into a message and generate an activity for a published message:

1. Create your propagator instances. The .NET PubSub+ API supports trace context and baggage propagation. If you plan to use both in your application, we recommend you use a CompositeTextMapPropagator which takes in a list of TextMapPropagators :

List<TextMapPropagator> propagators = new List<TextMapPropagator>() { new TraceContextPropagator(), new BaggagePropagator() };
CompositeTextMapPropagator compositeContextPropagator;
// ...				
compositeContextPropagator = new CompositeTextMapPropagator(propagators);

2. Create an instance of ActivitySource, which allows you to create and start activity objects. Use activitySource.StartActivity() to start the activity:

ActivitySource activitySource;
// ...
using var activity = activitySource.StartActivity("activityName", ActivityKind.Producer);	

3. (Optional) Attach baggage to a message with the Baggage class and the SetBaggage(key,value) method:

   Baggage.SetBaggage("myBaggageKey", "myBaggageValue");	 

4. Use your compositeContextPropagator to inject the context (and optional baggage) into a message:

   compositeContextPropagator.Inject(new PropagationContext(activity.Context, Baggage.Current), message, SolaceMessageCarrier.Setter);	

5. Send the message and call the Stop() method to export the activity data:

   session.Send(message);
   activity?.Stop();				

Extract Context from an Inbound Message

The following steps show you how to extract tracing context from a received message and generate an activity:

1. Create propagator and ActivitySource instances. The .NET PubSub+ API supports trace context and baggage propagation. If you plan to use both in your application, we recommend you use a CompositeTextMapPropagator which takes in a list of TextMapPropagators:

List<TextMapPropagator> propagators = new List<TextMapPropagator>() { new TraceContextPropagator(), new BaggagePropagator() };
CompositeTextMapPropagator compositeContextPropagator;
ActivitySource activitySource;    // Allows you to create and start Activity objects
// ...				
compositeContextPropagator = new CompositeTextMapPropagator(propagators);

2. In your receiver application's message handler, extract the context and baggage (if applicable) from a received message. Create a new ActivityContext object with the extracted message context (parentContext in the example below):

private void HandleMessageEvent(object source, MessageEventArgs args)
{
	using (IMessage message = args.Message)
	{
		if (message == null) return;
		var propagatorContext = compositeContextPropagator.Extract(default, message, SolaceMessageCarrier.Getter);
		ActivityContext parentContext = propagatorContext.ActivityContext;
		Baggage.Current = propagatorContext.Baggage;

3. Create and start a consumer activity object with activitySource.StartActivity():

   using var activity = activitySource.StartActivity("activityName", ActivityKind.Consumer, parentContext);			

4. Set the parent ID with the SetParentId() method:

   activity?.SetParentId(parentContext.SpanId.ToString());				

5. Call the Stop() method to export the activity data:

   activity?.Stop();			

Instrumenting JavaScript and Node.js for Distributed Tracing

Manual Instrumentation involves making changes to your enterprise application's source code, and allows you to inject and extract additional context, such as baggage and trace states, into messages. Context propagation makes it easy to debug and optimize your application. For more information about context propagation in Solace event messages, see Distributed Tracing Context Propagation. The following examples show you how to create spans using the OpenTelemetry API.

Understanding How Context Propagation Enables Distributed Tracing in the JavaScript and Node.js PubSub+ APIs

In your client application, you can use the OpenTelemetry API to create a span, which contains metadata about an operation in a distributed system. This span is associated with a context, which includes a unique TraceID. Next, when you use a PubSub+ message producer to publish a message, the Solace OTel integration package injects the context, which contains the TraceID, into the message. As the message travels through the event broker and is received by a consuming application, spans are generated at each step and have the same TraceID present in the original message context. When each span is closed in the publishing or consuming application, the JavaScript OpenTelemetry API sends it to an OpenTelemetry collector, which collects, processes and exports the spans to a backend application that correlates the spans using their unique TraceID. A backend application uses the correlated spans to create a trace, which is an end-to-end snapshot detailing how the message traveled through the distributed system. If you do not use context propagation, then backend applications cannot use a unique TraceID to link the spans, making it difficult to trace the flow of messages through the distributed system.

Dependencies

To enable context propagation for distributed tracing, you must first add the Solace OpenTelemetry JavaScript Integration artifact to your application. For OpenTelemetry version compatibility see Distributed Tracing Version Compatibility. Adding the libraries gives you access to the following two classes:

• SolaceW3CTextMapSetter
  • use with the OpenTelemetry API's W3CTraceContextPropagator to inject context into a Message.
  • use with the OpenTelemetry API's W3CBaggagePropagatorto inject baggage into a Message.
• SolaceW3CTextMapGetter
  • use with the OpenTelemetry API's W3CTraceContextPropagator to extract context from a Message.
  • use with the OpenTelemetry API's W3CBaggagePropagatorto extract baggage from a Message.

To use context propagation in the PubSub+ JavaScript API and PubSub+ Node.js API, include the following modules in your application:

import {
    SemanticAttributes,
    MessagingOperationValues,
    MessagingDestinationKindValues,
} from '@opentelemetry/semantic-conventions';

const opentelemetry = require('@opentelemetry/api');
const { CompositePropagator, W3CTraceContextPropagator, W3CBaggagePropagator, TraceState } = require("@opentelemetry/core");
const { BasicTracerProvider } = require("@opentelemetry/sdk-trace-base");
const { context, propagation, trace } = require("@opentelemetry/api");
const { SemanticAttributes, MessagingDestinationKindValues } = require("@opentelemetry/semantic-conventions");
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SolaceW3CTextMapSetter, SolaceW3CTextMapGetter } = require("pubsubplus-opentelemetry-js-integration");

const api = require('@opentelemetry/api'); 
var solace = require('solclientjs').debug; // with logging supported

Injecting Context into an Outbound Message

Your publishing application can generate a send span and export it to the OpenTelemetry Collector. The following steps show you how to inject context into a message and generate a send span for a published message:

1. Create propagator instances. The JavaScript and Node.js APIs supports trace context and baggage propagation. If you plan to use both in your application, we recommend you use a CompositePropagator which takes in an array of propagators:

   const compositePropagator = new CompositePropagator({
      propagators: [
        new W3CBaggagePropagator(),
        new W3CTraceContextPropagator(),
      ],
   });
   

2. Set the global propagator with a W3CTraceContextPropagator() and optional W3cCBaggagePropagator that ensures all context propagation in your application follows the same rules and mechanisms. In the example below, both are contained in the compositePropagator created in step 1. Next create a BasicTracerProvider which allows your application to create and activate traces:

propagation.setGlobalPropagator(compositePropagator);
const tracerProvider = new BasicTracerProvider();

3. Use the register() function to register the tracerProvider created in step 1 with the OpenTelemetry API. This lets you use tracer instances to create spans and record trace data. Next create a tracer. The parameter you pass to the getTracer() function represents the name of the tracer, which is used to associate the tracer with a specific component, service, or module within your application:

tracerProvider.register();
const tracer = opentelemetry.trace.getTracer("solace-pubsub-publisher-test");

4. Use the OpenTelemetry API to retrieve the currently active context that will be propagated. Then, create a "publish span" and start it with the startSpan() function:

   let ctx = api.context.active();
   const span = tracer.startSpan(topicName+' send',{kind: opentelemetry.SpanKind.CLIENT}, ctx);	 

5. Set span attributes using setAttribute() function. This lets you attach additional context and meta-data to a span. You can also use setAttribute() to label the span with a descriptive name that indicates the type of operation the span represents.

   span.setAttribute('attributeKey','attributeValue');
   span.setAttribute(SemanticAttributes.MESSAGING_OPERATION, 'send');
   

6. (Optional) Create baggage with the createBaggage() function and set it on the context by calling the setBaggage() function on your propagation instance:

   const baggage = propagation.createBaggage({
     "baggageKey": {
       value: 'baggageValue',
       metadata: undefined,
     },
   }); 
   ctx = propagation.setBaggage(ctx, baggage);		

7. Create an instance of SolaceContextSetter and use it with a W3C context propagator to inject context into a Solace message. Send the message and set the SpanStatusCode with the setStatus() function:

   var setter = new SolaceContextSetter();
   opentelemetry.propagation.inject(ctx,solaceMessage,setter)
   try {
       solaceSession.send(solaceMessage);
       span.setStatus({
           code: opentelemetry.SpanStatusCode.OK,
           message: 'Message Sent'
        });
   } catch (error) {
       span.setStatus({
           code: opentelemetry.SpanStatusCode.ERROR,
           message: error.toString()
       });
   }				

8. Call the end() function on the span to close it and export the data.

   span.end();		

Extracting Context from an Inbound Message

Your consuming application can generate a process span and then export it to the OpenTelemetry Collector. The following steps show you how to extract tracing context from a received message and generate a process span:

1. Use the OpenTelemetry API to create a tracer: The parameter you pass to the getTracer() function represents the name of the tracer, which is used to associate the tracer with a specific component, service, or module within your application:

const tracer = opentelemetry.trace.getTracer('solace-pubsub-receiver-test');

2. In your event listener for messages, create a SolaceContextGetter. Extract the trace and context information from a received message with the extract() function. Use this extracted context to create a parentContext for tracing subsequent operations:

messageConsumer.on(solace.MessageConsumerEventName.MESSAGE, function (message) {
    var getter = new SolaceContextGetter();
    const parentContext = propagation.extract(opentelemetry.ROOT_CONTEXT,message,getter);

3. Create a new process span and link it to the parentContext if it exists:

   if(!parentContext)
       var span = tracer.startSpan(MessagingOperationValues.PROCESS,{kind: opentelemetry.SpanKind.CLIENT});
   else
       var span = tracer.startSpan(MessagingOperationValues.PROCESS,{kind: opentelemetry.SpanKind.CLIENT},parentContext);		

4. Set span attributes using setAttribute() function. This lets you attach additional context and meta-data to a span. You can also use setAttribute() to label the span with a descriptive name that indicates the type of operation the span represents:

   span.setAttribute('attributeKey','attributeValue');
   span.setAttribute(SemanticAttributes.MESSAGING_OPERATION, 'process');
   

5. (Optional) Extract baggage from the parent context using the getBaggage() function:

const baggage = propagation.getBaggage(parentContext);	

6. Process the message and set and set the SpanStatusCode with the setStatus() function:

       try {
           // Process the message here
           span.setStatus({
               code: opentelemetry.SpanStatusCode.OK,
               message: 'Message processed'
           });		
           message.acknowledge();
       }  catch (error) {
           span.setStatus({
               code: opentelemetry.SpanStatusCode.ERROR,
               message: error.toString()
           });
       }
   

7. Call the end() function on the span to close it and export the data:

   span.end();