Context Propagation for Distributed Tracing in the Go API

Distributed tracing allows your enterprise applications to trace the flow of messages as they travel from your publisher, through the event mesh and to the receiving application. For a detailed overview see Distributed Tracing . For information about version requirements, see Distributed Tracing Version Compatibility.

Instrumenting Go for Distributed Tracing

Manual Instrumentation involves making changes to your enterprise application's source code, and allows you to inject and extract additional context, such as baggage and trace states, into messages. Context propagation makes it easy to debug and optimize your application. For more information about context propagation in Solace event messages, see Distributed Tracing Context Propagation. The following examples show you how to create spans using the OpenTelemetry API.

Understanding How Context Propagation Enables Distributed Tracing in the Go PubSub+ API

In your client application, you can use the OpenTelemetry API to create a span, which contains metadata about an operation in a distributed system. This span is associated with a context, which includes a unique TraceID. Next, when you use a PubSub+ message producer to publish a message, the Solace OTel integration package injects the context, which contains the TraceID, into the message. As the message travels through the event broker and is received by a consuming application, spans are generated at each step and have the same TraceID present in the original message context. When each span is closed in the publishing or consuming application, the Go OpenTelemetry API sends it to an OpenTelemetry collector, which collects, processes and exports the spans to a backend application that correlates the spans using their unique TraceID. A backend application uses the correlated spans to create a trace, which is an end-to-end snapshot detailing how the message traveled through the distributed system. If you do not use context propagation, then backend applications cannot use a unique TraceID to link the spans, making it difficult to trace the flow of messages through the distributed system.

Dependencies

To enable context propagation for distributed tracing, you must first add the Solace PubSub+ OpenTelemetry Integration For Solace Go API as a dependency in your application. You can also add this package with the following command:

go get solace.dev/go/messaging-trace/opentelemetry

Then add the OpenTelemetry API and SDK libraries required for context propagation with the following commands:

go get -u go.opentelemetry.io/otel
go get -u go.opentelemetry.io/otel/sdk	

For OpenTelemetry version compatibility, see Distributed Tracing Version Compatibility. Adding these libraries gives you access to the TextMapCarrier interface which gives your application access to:

• NewInboundMessageCarrier—contains all of the functions that the OpenTelemetry API needs to inject and extract traces and baggage in inbound messages.
• NewOutboundMessageCarrier—contains all of the functions that the OpenTelemetry API needs to inject and extract traces and baggage in outbound messages.

To use context propagation in the Go PubSub+ API, include the following packages in your application:

import (
       // ...
       // ...	
       "go.opentelemetry.io/otel/attribute"                      // Package for handling OpenTelemetry attributes	
       semconv "go.opentelemetry.io/otel/semconv/v1.19.0"        // Semantic conventions for OpenTelemetry	
       "go.opentelemetry.io/otel/trace"                          // Package for working with OpenTelemetry traces
       "go.opentelemetry.io/otel/baggage"	                 // (Optional) Package for working with OpenTelemetry baggage
       "go.opentelemetry.io/otel"                                // OpenTelemetry core library for instrumentation and APIs	
       "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"	 // An implementation of the console trace exporter
       sdktrace "go.opentelemetry.io/otel/sdk/trace"		 // Gain access to the tracing functionality provided by the OpenTelemetry SDK
       otel_propagation "go.opentelemetry.io/otel/propagation"   // Package for propagation of OpenTelemetry context. Use an alias to prevent conflicts.	
       propagation "solace.dev/go/messaging-trace/opentelemetry" // Solace PubSub+ Otel integration
)

Generating a Send Span on Message Publish

Your publishing application can generate a send span and export it to the OpenTelemetry Collector. The following steps show you how to inject context into a message and generate a send span for a published message:

1. Initialize and create an exporter, which allows you to export your data to an OpenTelemetry backend. Then, set up a traceProvider, which manages the lifecycle of tracers in your application. A traceProvider needs to be initialized before your application can start receiving valid spanContext data.

exporter, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())
batchspanProcessor := sdktrace.NewSimpleSpanProcessor(exporter) // You should use a batch span processor in production
traceProvider := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(sdktrace.AlwaysSample()),
    sdktrace.WithSpanProcessor(batchspanProcessor), 
)

2. Initialize and create TraceContext and optional Baggage propagator instances, and register them with the SetTextMapPropagator()function. This function makes the TraceContext and Baggage structures available across your application. Register your global tracer provider with the SetTracerProvider() function, which ensures all tracers created in your code use the settings specified in traceProvider:

otel.SetTextMapPropagator(otel_propagation.TraceContext{})
otel.SetTracerProvider(traceProvider)
								
// If you use Baggage propagation:
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(otel_propagation.TraceContext{}, otel_propagation.Baggage{}))
otel.SetTracerProvider(traceProvider)			

3. Create an instance of NewOutboundMessageCarrier, a struct that allows TraceContextand Baggage to be propagated.

outboundMessageCarrier := propagation.NewOutboundMessageCarrier(outboundMessage)	

4. Set span attributes using an attribute.KeyValue slice, which lets you attach additional context and meta-data to a span in the form of key-value pairs. Then use a trace.SpanStartOption slice to define the new span options:

attrsForPublishSpan := []attribute.KeyValue{
    semconv.MessagingSystem("PubSub+"),                 // Messaging system to use
    semconv.MessagingDestinationKindTopic,              // Publish to a topic
    semconv.MessagingDestinationName(publishToTopic),   // topic name
    semconv.MessagingOperationPublish,                  // message operation is 'publish'
    semconv.MessagingMessageID(messageID.String()),     // ID of message to publish
}
optsForPublishSpan := []trace.SpanStartOption{
    trace.WithAttributes(attrsForPublishSpan...),       // Include the span attributes
    trace.WithSpanKind(trace.SpanKindProducer),         // Sets the span kind as 'producer'
}

5. (Optional) Create Baggage objects with key-value pairs. Add the Baggage to a context instance:

contextWithBaggage:= context.Background()
m0, _ := baggage.NewMember(string("myKey1"), "value1")
m1, _ := baggage.NewMember(string("myKey2"), "value2")
b, _ := baggage.New(m0, m1)
contextWithBaggage = baggage.ContextWithBaggage(contextWithBaggage, b)		

6. Create and initialize a Tracer, which is required to start spans, and then use that tracer to start a new span. This span enables the tracing of the message publication process with associated meta-data and attributes.

publisherTracer := otel.GetTracerProvider().Tracer("myAppTracer", trace.WithInstrumentationVersion(Version()))
publishCtx, publishSpan := publisherTracer.Start(context.Background(), fmt.Sprintf("%s publish", publishToTopic), optsForPublishSpan...)
			
// If propagating baggage, pass in the contextWithBaggage object created in step 4:
publishCtx, publishSpan := publisherTracer.Start(contextWithBaggage, fmt.Sprintf("%s publish", publishToTopic), optsForPublishSpan...)			

7. Use defer to call the End() function on the publish span right after you start the span. This line of code always executes right before the surrounding function ends, even if your application panics or has multiple returns.

defer publishSpan.End()		

8. Inject the context into the current span:

otel.GetTextMapPropagator().Inject(publishCtx, outboundMessageCarrier)

9. Publish the message:

publishErr := persistentPublisher.Publish(outMessage, resource.TopicOf(publishToTopic), nil, nil)        
if publishErr != nil {
    panic(publishErr)
}			

For a complete example of distributed tracing in a message publisher, see publisher.go on the Solace Developers Hub.

Generating a Receive Span on Message Receive

Your consuming application can generate a receive span and then export it to the OpenTelemetry Collector. The following steps show you how to extract tracing context from a received message and generate a receive span:

1. Initialize and create an exporter, which allows you to export your data to an OpenTelemetry backend. Then, set up a traceProvider, which manages the lifecycle of tracers in your application. A traceProvider needs to be initialized before your application can start receiving valid spanContext data.

exporter, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())
batchspanProcessor := sdktrace.NewSimpleSpanProcessor(exporter) // You should use a batch span processor in production
traceProvider := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(sdktrace.AlwaysSample()),
    sdktrace.WithSpanProcessor(batchspanProcessor), 
)

2. Initialize and create TraceContext and optional Baggage propagator instances, and register them with the SetTextMapPropagator()function. This function makes the TraceContext and Baggage structures available across your application. Register your global tracer provider with the SetTracerProvider() function, which ensures all tracers created in your code use the settings specified in traceProvider:

otel.SetTextMapPropagator(otel_propagation.TraceContext{})
otel.SetTracerProvider(traceProvider)
								
// If you use Baggage propagation:
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(otel_propagation.TraceContext{}, otel_propagation.Baggage{}))
otel.SetTracerProvider(traceProvider)			

3. In a MessageHandler callback, create an instance of NewInboundMessageCarrier, a struct that allows TraceContextand Baggage to be propagated. Next, create your context instance, and use OpenTelemetry's GetTextMapPropagator() and Extract() functions to retrieve the trace context information from the NewInboundMessageCarrier. The extracted TraceContext can then be propagated with subsequent outbound messages. context.Background() creates a base context to which the extracted TraceContext is added.

var messageHandler solace.MessageHandler = func(message message.InboundMessage) {
    inboundMessageCarrier := propagation.NewInboundMessageCarrier(message)
    parentSpanContext := otel.GetTextMapPropagator().Extract(context.Background(), inboundMessageCarrier)			

4. (Optional) Extract any Baggage from the current span with the baggage.FromContext() function:

extractedBaggage := baggage.FromContext(parentSpanContext)

5. Set span attributes using an attribute.KeyValue slice, which lets you attach additional context and meta-data to a span in the form of key-value pairs. Then use a trace.SpanStartOption slice to define the new span options:

attrs := []attribute.KeyValue{
    semconv.MessagingSystem("PubSub+"),                               // Messaging system to use
    semconv.MessagingDestinationKindTopic,                            // Use topic subscriptions
    semconv.MessagingDestinationName(message.GetDestinationName()),   // topic name
    semconv.MessagingOperationReceive,                                // message operation is 'Receive'
    semconv.MessagingMessageID(messageID.String()),                   // ID of message to publish
}
opts := []trace.SpanStartOption{
    trace.WithAttributes(attrs...),                 // Include the span attributes
    trace.WithSpanKind(trace.SpanKindConsumer),     // Sets the span kind as 'Consumer'
}			

6. Create and initialize a Tracer, required to start spans, and then use that tracer to start a new span. This span enables the tracing of the message process with associated meta-data and attributes.

tracer := otel.GetTracerProvider().Tracer("myAppTracer", trace.WithInstrumentationVersion(Version()))
receiveCtx, span := tracer.Start(parentSpanContext, fmt.Sprintf("%s receive", message.GetDestinationName()), opts...)	

7. Use defer to call the End() function on the receive span right after you start the span. This line of code always executes right before the surrounding function ends, even if your application panics or has multiple returns. With PersistentReceiverAutoAck enabled, the default setting in the Go API, messages are acknowledged after the End() function returns.

defer span.End()		

8. Process the received message:

var messageBody string
if payload, ok := message.GetPayloadAsString(); ok {
    messageBody = payload
} else if payload, ok := message.GetPayloadAsBytes(); ok {
    messageBody = string(payload)
}
fmt.Printf("Received Message Body %s \n", messageBody)

For a complete example of distributed tracing in a message subscriber, see subscriber.go on the Solace Developers Hub.