Distributed Tracing Example

The following sections provide an example of how you can use distributed tracing to help speed up troubleshooting in your network.

This example shows you how to configure initial telemetry settings, then enable and disable tracing for two types of traffic in an order management system.

Before You Begin

Before you start this example, you must have:

a deployed receiver, as described in Solace OpenTelemetry Receiver
a configured Message VPN (named myVpn in this example)

Step 1: Configure Initial Telemetry Settings

In this example, the first step is to configure the event broker so that receiver clients can bind to the telemetry data queue.

To configure the initial telemetry settings:

Create a telemetry profile for the Message VPN.

solace> enable
solace# configure
solace(configure)# message-vpn myVpn
solace(configure/message-vpn)# create telemetry-profile default
solace(configure/message-vpn/telemetry-profile)# end

Create a client username.

solace# configure
solace(configure)# create client-username telemetry-receiver message-vpn myVpn

Bind the client username to the telemetry profile's client profile and ACL profile objects, which is required to be able to bind to and consume from the telemetry profile's data queue.
```
solace(configure/client-username)# acl-profile #telemetry-default
solace(configure/client-username)# client-profile #telemetry-default
```
Assign a password if the client username will use basic authentication, then enable the client username.
```
solace(configure/client-username)# password *******
solace(configure/client-username)# no shutdown
```

Step 2: Configure Tracing

Next, we will enable tracing, and set up trace filters to identify messages that should be traced.

In this example, there are two types of traffic we want to trace:

An order management system's messages, where all messages are to be tracked as they move through the event mesh. Generally, we always want to trace these messages.
Some logging-related messages, where the primary use case to trace these messages is for debugging (when messages are not getting to their desired destination). We don't want tracing on for these messages all the time, but only when a debugging activity is in progress.

To configure tracing:

Enable tracing. This won't actually do anything yet without any enabled trace filters, but it allows tracing to begin when a trace filter is created and enabled.

solace> enable
solace# configure
solace(configure)# message-vpn myVpn
solace(configure/message-vpn)# telemetry-profile default
solace(configure/message-vpn/telemetry-profile)# trace
solace(configure/message-vpn/telemetry-profile/trace)# no shutdown

Create the OrderMgmtSystem trace filter with all messages using the first topic level of oms, unless the fourth topic level is pii.

solace(configure/message-vpn/telemetry-profile/trace)# create filter OrderMgmtSystem
solace(configure/message-vpn/telemetry-profile/trace/filter)# create subscription oms/>
solace(configure/message-vpn/telemetry-profile/trace/filter/subscription)# exit
solace(configure/message-vpn/telemetry-profile/trace/filter)# create subscription !oms/*/*/pii/>
solace(configure/message-vpn/telemetry-profile/trace/filter/subscription)# exit

Enable the OrderMgmtSystem trace filter to begin tracing on these messages.

solace(configure/message-vpn/telemetry-profile/trace/filter)# no shutdown
solace(configure/message-vpn/telemetry-profile/trace/filter)# exit

Create the LogFilter trace filter, but don't enable it yet.

solace(configure/message-vpn/telemetry-profile/trace)# create filter LogFilter
solace(configure/message-vpn/telemetry-profile/trace/filter)# create subscription log/>
solace(configure/message-vpn/telemetry-profile/trace/filter/subscription)# exit
solace(configure/message-vpn/telemetry-profile/trace/filter)# create subscription !log/*/*/pii/>

Step 3: Enable a Debugging Activity

With the configuration above in place, suppose we want to begin a debugging activity using logging-related messages.

To start tracing these logging-related messages, enable the LogFilter trace filter:

solace> enable
solace# configure
solace(configure)# message-vpn myVpn
solace(configure/message-vpn)# telemetry-profile default
solace(configure/message-vpn/telemetry-profile)# trace filter LogFilter
solace(configure/message-vpn/telemetry-profile/trace/filter)# no shutdown

Step 4: Disable Tracing

Perhaps the system is getting overwhelmed by the amount of tracing it is doing, which is adversely affecting other traffic. Rather than spending time to selectively enable and disable individual trace filters, we want to stop tracing now.

To disable tracing, enter the following commands:

solace> enable
solace# configure
solace(configure)# message-vpn myVpn
solace(configure/message-vpn)# telemetry-profile default
solace(configure/message-vpn/telemetry-profile)# trace shutdown

Provide feedback