Configuring Distributed Tracing

When distributed tracing is enabled, the event broker generates special guaranteed messages when certain operations happen. A purpose-built client (an instance of the Solace Open Telemetry Receiver) then consumes these messages from a particular queue and converts them to a format that can be received, processed, and viewed using common Open Telemetry backends such as Jaeger and DataDog. For more information, see Distributed Tracing .

To configure distributed tracing, follow these steps:

  1. Create a telemetry profile for the Message VPN.
  2. Create a client username.
  3. Bind the client username to the client profile and the ACL associated with the telemetry profile.
  4. Configure and enable tracing.

The following sections describe how to configure a telemetry profile and enable telemetry tracing. For a detailed example that walks through all the steps required to set up a working distributed tracing scenario, see Distributed Tracing Example.

Configuring Telemetry Profiles

On the event broker, distributed tracing is enabled by a telemetry profile, which is configured at the message VPN level. You can give the telemetry profile any name. When you create a telemetry profile (with a name, <profileName>) for a message VPN, the event broker automatically creates the following objects:

  • a special-purpose queue, named #telemetry-<profileName>. This queue is used only for telemetry messages. For details about configuring this queue, see Configuring Telemetry Queue Settings.
  • a client profile, named #telemetry-<profileName>, that controls certain configuration settings for telemetry receivers (or simply receivers). Receivers are clients that bind to the telemetry queue and consume trace messages. To obtain the receiver configuration from the telemetry profile, this client profile must be assigned to the client username or authorization group object the receiver uses for authorization. An attempt to bind to the telemetry queue from a client using a different client profile will fail. Note that the telemetry ACL profile is the object that explicitly allows binding to the telemetry queue, but an additional check is made to ensure the ACL profile name and client profile name match in order to prevent accidental misconfiguration.
  • an ACL profile, also named #telemetry-<profileName>, that controls whether clients can bind to the profile's corresponding telemetry queue. For a client to be permitted to bind to the telemetry queue, its ACL profile name must match the telemetry queue name during bind.

There can be only one telemetry profile per Message VPN.

The telemetry profile also has a trace property that you configure to control the nature of the data that goes into the queue, and a receiver property to control how data gets out of the queue. For details about these properties, see Configuring Trace Filters and Enabling Tracing and Configuring Receiver Client Connection Settings.

To create a telemetry profile, enter the following commands: 

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# create telemetry-profile <telemetry-profile-name>

To configure an existing telemetry profile:

solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>

Where:

<message-vpn-name> is the name of the Message VPN where you want to create the telemetry profile.

<telemetry-profile-name> is the name of the telemetry profile.

The configuration tasks you can perform for a telemetry profile include:

Configuring Telemetry Queue Settings

A telemetry queue is a special-purpose queue, used only for telemetry messages.

The configuration tasks you can perform for a telemetry queue include:

Configuring the Maximum Number of Flows that May Bind to the Telemetry Queue

To configure the maximum number of flows that may bind to the telemetry queue, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# queue
solace(...e/message-vpn/telemetry-profile/queue)# max-bind-count <value>

Where:

<value> is the integer value specifying the maximum number of flows that can bind the queue. The valid range is 0 to 10,000. The default value is 1,000.

The no version of this command, no max-bind-count, returns the value to the default.

Configuring Maximum Spool Usage Values for the Telemetry Queue

To configure the maximum amount of message spool that the telemetry queue may use, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# queue
solace(...e/message-vpn/telemetry-profile/queue)# max-spool-usage <size>

Where:

<size> is an integer value specifying the maximum amount of message spool disk space permitted for the queue in MB.

The no version of this command, no max-spool-usage, resets the permitted message spool usage quota available for use by the queue back to the default.

Configuring Event Thresholds for the Telemetry Queue

Event thresholds control when various events are generated by the event broker. You can configure the following event thresholds for a telemetry queue:

  • Bind Count Thresholds
  • Spool Usage Thresholds

To configure the event thresholds for a telemetry queue, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# queue
solace(...e/message-vpn/telemetry-profile/queue)# event

The CLI is now at a level at which you can configure the bind count or spool usage set and clear thresholds for the telemetry queue. For more information about queue event thresholds, see Queue Event Thresholds.

Configuring Receiver Client Connection Settings

A receiver is a client application that binds to the telemetry queue and consumes trace messages. For more information, see Solace Open Telemetry Receiver.

The configuration tasks you can perform for receiver client connections include:

Configuring the Maximum Number of Receiver Client Connections Per Client Username

To configure the maximum number of receiver client connections per client username, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# receiver
solace(...essage-vpn/telemetry-profile/receiver)# max-connections-per-client-username <value>

Where:

<value> is an integer value specifying the maximum number of simultaneous client connections that are permitted.

The no version of this command, no max-connections-per-client-username, resets the maximum number of client connections back to the default value.

Configuring Event Thresholds for Receiver Client Connections

Event thresholds control when various events are generated by the event broker. You can configure the following event thresholds for receiver client connections:

  • Connections Per Client Username Thresholds

To configure event thresholds for receiver client connections, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# receiver
solace(...essage-vpn/telemetry-profile/receiver)# event

The CLI is now at a level at which you can configure the set and clear thresholds. For more information about client event thresholds, see Configuring Client Event Thresholds.

Configuring TCP Settings for Receiver Client Connections

To configure the TCP keepalive for receiver client connections, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# receiver
solace(configure/message-vpn/telemetry-profile/receiver)# tcp
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# keepalive
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# count <num>
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# idle <seconds>
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# interval <seconds>

Where:

count <num> sets the maximum number of keepalive probes (from 2 to 5 ) TCP should send before dropping the connection. The default value is 5.

idle <seconds> sets the time (from 3 to 120 seconds) a connection must remain idle before TCP begins sending keepalive probes. The default value is 3.

interval <seconds> sets the time (from 1 to 30 seconds) to set as the interval between individual keepalive probes. The default value is 1.

The no version of these commands resets the values to back to the defaults.

To configure initial congestion window sizes for receiver client connections, enter the following commands:

solace(configure/message-vpn/telemetry-profile/receiver)# tcp
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# initial-cwnd <num-mss>

Where:

initial-cwnd <num-mss> is an integer specifying the size of the TCP initial congestion window measured in number of TCP maximum segment size (MSS). Valid values are 2 through 7826. The default value is 2.

To configure max window sizes for receiver client connections, enter the following commands:

solace(configure/message-vpn/telemetry-profile/receiver)# tcp
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# max-wnd <num-kilo-bytes>

Where:

max-wnd <num-kilo-bytes> is an integer specifying the size of the TCP maximum window measured in number of kilobytes (KB). Valid values are 32 through 65536. The default value is 256.

To configure max segment sizes for receiver client connections, enter the following commands:

solace(configure/message-vpn/telemetry-profile/receiver)# tcp
solace(configure/message-vpn/telemetry-profile/receiver/tcp)# mss <byte-count>

Where:

mss <byte-count> is the size of the maximum segment size in bytes (256 through 1460). The default value is 1460.

Enabling or Disabling Receiver Client Connections

To enable connecting receiver clients to connect to and consume from the telemetry queue, enter the following command:

solace(configure/message-vpn/telemetry-profile/receiver)# no shutdown

To disable connecting receiver clients from connecting to and consuming from the telemetry queue, enter the following command:

solace(configure/message-vpn/telemetry-profile/receiver)# shutdown

Configuring ACL Settings for Receiver Clients

You can use ACLs to control which receiver clients may connect to the event broker.

To set the default action for receiver client connection attempts to the event broker, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# receiver
solace(...essage-vpn/telemetry-profile/receiver)# acl
solace(...ge-vpn/telemetry-profile/receiver/acl)# connect
solace(...elemetry-profile/receiver/acl/connect)# default-action {allow | disallow}

Where:

allow sets the default action to allow receiver client connection attempts.

disallow sets the default action to disallow receiver client connection attempts. This is the default value.

To set exceptions to the default action for receiver client connection attempts, enter the following commands:

solace(...elemetry-profile/receiver/acl/connect)# exception <cidr-addr>

Where:

exception <cidr-addr> is the IP address and network mask combination of the excepted client in Classless Inter-Domain Routing (CIDR) form: nnn.nnn.nnn.nnn/dd (where nnn is 0-255, dd is 1-32).

Changing the default client connect action, or removing clients from the exceptions list, does not immediately affect clients that already have an established connection to the event broker. They remain connected.

Configuring Trace Filters and Enabling Tracing

When you enable tracing for a telemetry profile, the event broker generates spans at certain points in the lifecycle of an event message. A span represents a single operation on the event broker, such as receiving a message or acknowledging a message.

To configure tracing, follow these steps:

  1. Create a trace filter
  2. Configure subscriptions for the filter
  3. Enable the trace filter
  4. Enable tracing

Configuring Trace Filters

You use a trace filter to group together subscriptions that fulfill a particular tracing use case (in other words, subscriptions that you would want to enable or disable together). If an application's messages can be described by a set of subscriptions, then grouping those subscriptions into a trace filter would allow you to start or stop debugging simply by enabling or disabling that single trace filter.

To create a trace filter, enter the following commands:

solace(configure)# message-vpn <message-vpn-name>
solace(configure/message-vpn)# telemetry-profile <telemetry-profile-name>
solace(configure/message-vpn/telemetry-profile)# trace
solace(configure/message-vpn/telemetry-profile/trace)# create filter <filter-name>

Where:

<filter-name> is the name used to identify the trace filter.

The no version of this command, no filter <filter-name>, removes the trace filter.

Configuring Trace Subscriptions

Trace subscriptions control which messages received by the broker will be traced. If an incoming message matches an enabled trace filter's subscription, the message will be traced as it passes through the broker.

To configure a subscription for a trace filter, enter the following commands:

solace(...ge-vpn/telemetry-profile/trace/filter)# create subscription <subscription> [smf | mqtt]

Where:

<subscription> is the name of the topic subscription to be added to the filter in the form a/b/c.

smf specifies that the subscription uses SMF syntax. This is the default setting. For more information about SMF topic syntax, see Solace Message Format Topics.

mqtt specifies that the subscription uses MQTT syntax. For more information, see MQTT Topic Structure and Syntax.

Trace subscriptions are not exported. For more information, see Enabling Subscription Export.

Enabling or Disabling Trace Filters

To enable a trace filter, enter the following commands:

solace(...ge-vpn/telemetry-profile/trace/filter)# no shutdown

To disable a trace filter, enter the following commands:

solace(...ge-vpn/telemetry-profile/trace/filter)# shutdown

Enabling or Disabling Tracing

Typically you would leave tracing enabled, but enable or disable trace filters to control what gets traced.

To enable tracing for a Message VPN, enter the following commands:

solace(...ge-vpn/telemetry-profile/trace)# no shutdown

To disable tracing, enter the following commands:

solace(...ge-vpn/telemetry-profile/trace)# shutdown