Enabling Distributed Tracing

When distributed tracing is enabled, the event broker service generates trace messages when certain operations happen. A purpose-built client then consumes these messages from the telemetry queue and converts them to the OpenTelemetry standard format that can be received, processed, and viewed in PubSub+ Insights (requires an PubSub+ Insights account), or various common tracing backends supporting OpenTelemetry protocols, such as Datadog, Jaeger, DynaTrace, New Relic, and Grafana, among others.

You can install a distributed tracing license, create a tracing profile, and a tracing client all from the PubSub+ Cloud and PubSub+ Broker Manager. During this process, a pod is created that contains the Collector and Receiver. The Mission Control Agent creates the necessary telemetry queue, client username, and password for your event broker services.

To configure distributed tracing for your event broker service, follow these steps:

  1. Ensure your deployment meets the prerequisites for enabling distributed tracing.
  2. Create a tracing destination for the event broker service in the Cloud Console. See Creating a Tracing Destination.
  3. Create a telemetry profile for the event broker service in Broker Manager. See Creating a Telemetry Profile.
  4. Enable distributed tracing for your event broker service in the Cloud Console. See Enabling Distributed Tracing for an Event Broker Service.

This process can also be performed using the Solace CLI. See Using the Solace CLI to Enable Distributed Tracing for an Event Broker Service

Prerequisites for Enabling Distributed Tracing in PubSub+ Cloud

To use distributed tracing with your event broker services, you must meet these minimum requirements:

Infrastructure Impacts on the Performance of Distributed Tracing

The performance of distributed tracing in PubSub+ Cloud is affected by your choice of Kubernetes provider, and OpenTelemetry backend. Performance is tied to the ability of the receiver to consume and the collector to process guaranteed messages.

On average, you can expect a performance of at least 1000 trace spans per second. Different factors, such as your Kubernetes environment and settings, can affect these results.

Viewing Your Distributed Tracing Licenses

You must have a license for the event broker service on which you intend to enable distributed tracing.

Each license has a corresponding connection tier that indicates the largest service class of event broker service it can be used with. For example, a distributed tracing license with a connection tier of 10K can be used for event broker services ranging from 1,000 up to 10,000 connections. See Service Class Options for Event Broker Services for more information about service classes.

To get licenses for distributed tracing, contact Solace.

Screenshot showing the elements described in the surrounding text.

  • From the Cloud Console, click User & Account, select Account Details, and then click the Distributed Tracing tab.

    • The table located at the top contains the following information about your distributed tracing limits:

    • Connection Tier—The distributed tracing license tier.
    • In Use—The number of licenses currently being used by your event broker services.
    • Available—The number of licenses currently available to your event broker services.
    • Limits—The number of event broker services for which you have a distributed tracing license. This number is the sum of the number of licenses shown in the In Use and Available columns.

Creating a Tracing Destination

The first step in deploying distributed tracing for your event broker service is to create a tracing destination. The tracing destination configures your collector to send messages it receives from the receiver to your chosen tracing destination. You can create up to 10 tracing destinations. However, you can assign each event broker service a single tracing destination. This allows your event broker services to send data to different observability backends. When creating a tracing destination, you have several destination options, including:

The PubSub+ Insights option requires a separate PubSub+ Insights account and offers viewing of trace data in Insights without requiring additional configuration.

The other options on the list provide the ability to send trace data to the observability backends of your choice with some additional configuration. The OTLP options support sending your trace data using either gRPC or HTTP format, providing compatibility with a wide variety of observability backends, including Jaeger, DynaTrace, New Relic, and Grafana, among others. Observability backends may support one or both formats. When selecting a OTLP based option, you must select the format type and configure it so that it matches the format (HTTPS or gRPC) supported by your chosen observability backend. Refer to the documentation for your observability tool for information about the formats they support.

If you are deploying distributed tracing in a Customer-Controlled Region with a proxy, you do not need to perform any additional configuration, other than to ensure your proxy does not block the tracing headers. The distributed tracing collector gets your proxy configuration from the Mission Control Agent during initialization. See Using Distributed Tracing with a Proxy in a Customer-Controlled Region.

To create a tracing destination, perform these steps:

  1. Log in to the PubSub+ Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the PubSub+ Cloud Console.

  2. Click User & Account, select Account Details, and then click the Distributed Tracing tab.

    Screenshot showing the elements described in the surrounding text.

  3. Click Create Tracing Destination to open the Create Tracing Destination dialog and use the fields on the information related to your tracing destination. Note that the fields change depending on the destination type you select.
    • Name—enter a name for the tracing destination.
    • Type—select the service type and use the table below to guide your configuration options.

      • Destination TypeFields to configure
      PubSub+ Insights

      No additional configuration is required (requires a PubSub+ Insights account).

      Datadog
      • Region—Select your Datadog region from the drop-down menu.

        Your Datadog region is related to the URL you use to access your Datadog account. If you do not know your Datadog region, you can refer the table in the Datadog documentation to determine the region from the URL used to access your account.

      • API Key—Enter your Datadog API key into the field.
      OTLP/gRPC endpoint

      • Connection Type—Select your connection type from the drop-down menu. You can choose secure (skips certificate authority validation) or insecure.

        Note: If your endpoint uses HTTPS and a self-signed certificate, you must choose Secure (skips certificate authority validation) for the connection type.

      • Endpoint—Enter the host:port information for where the exporter will send the trace information using the protocol (HTTP or gRPC) supported by the observability backend. Refer to the documentation for your chosen observability backend for endpoint information and protocol support.

      • Header—Enter the credentials required by your observability backend to send them your tracing data.

      OTLP/HTTP endpoint

  4. Click Save.

  5. Click Create.

    The Distributed Tracing tab displays your tracing destination:

    Screenshot showing the elements described in the surrounding text.

  6. Optional: You can add additional tracing destinations by clicking Create Tracing Destination and repeating this procedure. You can create a maximum of 10 tracing destinations.

Editing a Tracing Destination

You can edit your tracing destination if required. Note that updating your tracing destination does not update the distributed tracing service on the event broker services using that tracing destination automatically. You must update the event broker service from the Services Requiring Updates window. See Updating the Tracing Destination.

  1. From the Cloud Console, click User & Account, select Account Details, and then click the Distributed Tracing tab.
  2. Click the ellipsis associated with the tracing destination you want to edit and select Edit to open the Edit Tracing Destination dialog.
  3. Change the fields on the window as needed and click Update.

    You must now update your event broker services to use the updated tracing destination. See Updating the Tracing Destination.

Updating the Tracing Destination

You can review and update the tracing destination for your event broker services as needed.

  1. From the Cloud Console, click User & Account, select Account Details, and then click the Distributed Tracing tab.

    2. The notification banner under the Tracing Destination title indicates the number of event broker services that require updating, if any.

    Screenshot showing the elements described in the surrounding text.

  2. Click View Services to open the Services Requiring Updates dialog.

    Screenshot showing the elements described in the surrounding text.

  3. Click Update on any event broker service that shows Update to upgrade the tracing destination for that event broker service. From this dialog you can also click on the ellipsis of any service and then click Service Details to open the Status tab of your service where you can view detailed service information, including whether you have deployed a license for distributed tracing to the service.

Deleting a Tracing Destination

You can delete a tracing destination you have created. You should note that deleting a tracing destination will cause any enabled distributed tracing configurations on event broker services to fail.

  1. From the Cloud Console, click User & Account, select Account Details, and then click the Distributed Tracing tab.
  2. Click the ellipsis associated with the tracing destination you want to delete and select Delete.

Creating a Telemetry Profile

The telemetry profile provides the information that your distributed tracing configuration requires to deploy the Collector and Receiver for your event broker service.

  1. From the Cluster Manager in the Cloud Console, select an existing service, then click More Actions and select Open PubSub+ Broker Manager.

    Screenshot showing the elements described in the surrounding text.

  2. In PubSub+ Broker Manager, click the Telemetry tab in the navigation bar and then click Create Telemetry Profile to open the New Telemetry Profile page.

    Screenshot showing the elements described in the surrounding text.

  3. Fill the following fields on the New Telemetry Profile page:
    • Telemetry Profile—Enter a name for the telemetry profile.
    • Trace—Click to set to Enabled.
    • Receiver—Click to set to Enabled.

    Optionally, fill the queue-related fields:

    • Maximum Consumer Count—Define the maximum number of consumer flows that can bind to the queue.
    • Message Queued Quota—Define the maximum message spool usage allowed by the queue, in megabytes (MB).

    Optionally, click Show Advanced Settings to access settings such as alert thresholds and TCP settings for the receiver and alert thresholds for the queue.

  4. When you have filled the appropriate fields, click Apply to create the telemetry profile and return to the Telemetry Profile page.

    Screenshot showing the elements described in the surrounding text.

  5. Click the Trace Filters tab and then click + Trace Filters to open the Create Trace Filters dialog.

    Screenshot showing the elements described in the surrounding text.

  6. In the Trace Filter Name field, enter a name for the trace filter and click Create.

    The Edit Trace Filter Settings page opens.

  7. Click the Enabled toggle to enable the trace filter and then click Apply.

    Screenshot showing the elements described in the surrounding text.

    You return to the Telemetry Profile page and Yes appears in the Enabled column for the trace filter you just created.

  8. Click the trace filter to select it, select the Subscriptions tab, and then click + Subscription.

    Screenshot showing the elements described in the surrounding text.

    The Create Subscriptions dialog opens.

    Screenshot showing the elements described in the surrounding text.

  9. In the Subscriptions field, enter the subscriptions you want this trace filter to be subscribed to. You can add multiple subscriptions by pressing enter after each entry. You can optionally use the drop-down list to specify the protocol for each subscription—either SMF (Solace Message Format) or MQTT.
  10. After adding your subscriptions, click Create.

    You return to the Subscriptions tab.

  11. Click the back arrow to return to the Trace Filters tab of the Telemetry Profiles page.

    Screenshot showing the elements described in the surrounding text.

  12. Select the Receiver Connect ACLs tab and click Edit.

    Screenshot showing the elements described in the surrounding text.

    The Edit Receiver ACL Connect Settings page opens.

  13. In the Client Connect Default Action list, select Allow.

    Screenshot showing the elements described in the surrounding text.

  14. Click Apply.

    15. You return to the Telemetry Profile page and your telemetry profile is ready to use once you enable distributed tracing on your event broker service. See Enabling Distributed Tracing for an Event Broker Service for more information.

Editing a Telemetry Profile

You can edit the various components of a telemetry profile that you have created. This does not affect the behavior of existing event broker services that have deployed distributed tracing using that telemetry profile. If you want the changes that you make to affect existing event broker services with distributed tracing, you must redeploy distributed tracing for the event broker services using the updated telemetry profile. See Enabling Distributed Tracing for an Event Broker Service for details.

  1. From the Cluster Manager in the Cloud Console, select an existing service. Click Open PubSub+ Broker Manager and select the Telemetry tab.
  2. Using the Settings, Receiver Connect ACLs, and Trace Filters tabs, navigate to the component you want to edit and click Edit .
  3. Make the changes that you want and click Apply.

Deleting a Telemetry Profile

You can delete a telemetry profile you have created. You should note that deleting a telemetry profile does not affect any event broker services that had deployed distributed tracing using that telemetry profile, but future distributed tracing deployments cannot use that telemetry profile. This action is permanent and cannot be reversed.

  1. From the Cluster Manager in the Cloud Console, select an existing service.
  2. Click Open PubSub+ Broker Manager and select the Telemetry tab.
  3. Click Action and select Delete.

    A confirmation window appears.

  4. Click OK.

Enabling Distributed Tracing for an Event Broker Service

After you have created a tracing destination and telemetry profile, you are ready to enable distributed tracing for your event broker service.

  1. From the Cluster Manager in the Cloud Console, select an existing service.
  2. Select the Manage tab and select Advanced Options.

    3. Screenshot showing the elements described in the surrounding text.

  3. Scroll to Distributed Tracing and click Enable. A message appears, indicating that a license is being deployed to the event broker service. After the operation completes, Enabled appears and a Deploy Configuration button appears.

    Screenshot showing the elements described in the surrounding text.

  4. Click Deploy Configuration to open the Configure Data Collection dialog.

    Screenshot showing the elements described in the surrounding text.

  5. Click Confirm and Deploy. Note that if you have multiple tracing destinations created, you must use the Tracing Destination field to select the tracing destination for the event broker service and then click Configure and Deploy. A tracing destination and telemetry profile must already exist. See Creating a Tracing Destination and Creating a Telemetry Profile for details.

    You return to the Manage tab and a Pending status appears beside Step 2: Trace Collection. The configuration can take up to several minutes to complete and will remain in a Pending state until finalized. The status changes to Active after configuration is complete.

    Screenshot showing the elements described in the surrounding text.

    6. You should note that enabling Distributed Tracing creates a new sc-dt-trace-collector client username for your event broker service. You can view the new client username in PubSub+ Broker Manager on the Clients and Access Control tabs.

    Do not delete or edit the client username. Doing so will cause your distributed tracing configuration to fail. See Using Client Profiles and Client Usernames for more information.

    You can disable distributed tracing for an event broker service. See Disabling Distributed Tracing.

Using Distributed Tracing with a Proxy in a Customer-Controlled Region

You can use distributed tracing with a proxy in a Customer-Controlled Region. This allows you to monitor the messages that travel through your estate while benefiting from the enhanced security features a proxy provides for your deployment. Using distributed tracing with a proxy is a hands-off experience: When you enable distributed tracing, the distributed tracing collector gets your proxy configuration from the Mission Control Agent during initialization. Besides meeting the prerequisites listed below, no additional configurations outside of the those detailed in this article are required.

Prerequisites for Using a Proxy in a Customer-Controlled Region

To use distributed tracing with a proxy in your Customer-Controlled Region, you must configure the proxy in your environment so that it does not block the tracing headers. This ensures the distributed tracing spans reach their destination.

Redeploying the Collectors Due to Proxy Changes

If either of the following situations occur, you must redeploy the collectors by updating your distributed tracing configuration:

  • If any changes are made to the proxy configurations contained in the Mission Control Agent

  • If you remove the proxy from your deployment.

To update your distributed tracing configuration see Checking Status and Updating Distributed Tracing.

Checking Status and Updating Distributed Tracing

You can check the status of distributed tracing on an event broker service. Checking the status allows you to troubleshoot any issues with distributed tracing function or performance. You can also update the distributed tracing configuration for the event broker service from the same window.

  1. From the Cluster Manager in the Cloud Console, select an existing service.
  2. Select the Manage tab and select Advanced Options.
  3. Scroll to Distributed Tracing and click Check Status. The status appears beside Step 2: Trace Collection. In the example below, the status is Pending.

    4. Trace collection can have one of the following statuses:

    • Active—Trace collection is active and trace events are being collected and delivered to your OpenTelemetry backend.
    • Pending—Trace collection is pending. The status of distributed tracing is being queried or changes are being applied.
    • Failed—Trace collection is not occurring. This status occurs for one or more of the following reasons:
      • The collector is down.
      • The event broker service cannot be found.
      • There was a timeout trying to connect to the event broker service.
      • A connection cannot be made to the event broker service.
  4. Optionally, you can update the distributed tracing configuration for the event broker service. If you choose to update the distributed tracing configuration for an event broker service, it updates all distributed tracing information, including any changes made to the telemetry profile and tracing destination. Updating the configuration may also resolve any failed status checks.

    To update the distributed tracing configuration, click Update Configuration to open the Configure Data Collection dialog. If you have more than one tracing destination configured for the account, the Tracing Destination field shows the tracing destination assigned to the event broker service. You can change the tracing destination by selecting a different one from the drop-down list..

    Screenshot showing the elements described in the surrounding text.

  5. Click Confirm and Update.

    6. The status beside Step 2: Trace Collection displays Pending. When the update is complete, the status changes to one outlined in step 3. If the tracing status continues to return a failed state, contact Solace for support.

Disabling Distributed Tracing

You can disable distributed tracing for an event broker service. Doing so releases the distributed tracing license so another event broker service can use it. It also releases the collector, removing the telemetry profile you created for the event broker service. It does not affect the tracing destination.

  1. From the Cluster Manager in the Cloud Console, select an existing service.
  2. Select the Manage tab and select Advanced Options.
  3. Scroll to Distributed Tracing and click Disable.

    A message appears indicating that the license is being redeployed. When the disable operation is complete, an Enable button appears.

Using the Solace CLI to Enable Distributed Tracing for an Event Broker Service

You can configure and enable distributed tracing for event broker services using the Solace CLI if you have it enabled. For details see Enabling the Solace CLI for Event Broker Services in PubSub+ Cloud.

To get started, you must log in to the CLI using a terminal like PuTTY and provide the following commands:

SolaceCLI > enable
SolaceCLI # configure

The remaining steps are identical to those used to configure distributed tracing for a software event broker. See Configuring Distributed Tracing for information.

You should note that while event broker services in PubSub+ Cloud are scoped to only one message VPN, you must still provide the Message VPN name when using the Solace CLI to set up distributed tracing for the event broker service. You can locate the name of the Message VPN for your event broker service by selecting your service from Cluster Manager and looking in the DMR Cluster section of the Status tab.

Screenshot showing the elements described in the surrounding text.

For instructions, see Configuring Distributed Tracing.