Extending Advanced Monitoring

Solace provides PubSub+ Insights dashboards for Datadog, PubSub+ Insights monitors for Datadog, and metrics that are valuable for general use cases whether you're in an operations and integration, or applications support team. The PubSub+ Insights dashboards are based on core best practices from subject-matter-experts and feedback from the Solace community. You can extend the Insights Advanced Monitoring capabilities to adapt them to match your specific use cases or requirements.

Users with the Insights Advanced Manager or Insights Advanced Editor role can do the following to customize the PubSub+ Insights dashboards and PubSub+ Insights monitors:

  • Create your own PubSub+ Insights dashboards and then copy widgets from an existing dashboard to add to yours. For more information, see the New Dashboard in the Datadog documentation.
  • Clone any of the PubSub+ Insights dashboards as a starting point and then add and remove widgets to suit your monitoring requirements. For more information, see Cloning Existing PubSub+ Insights Dashboards for Datadog for Reuse
  • Create custom views, add custom or existing PubSub+ Insights monitors, add custom widgets, customize existing widgets, and create filters on PubSub+ Insights dashboards you create or clone. For more information, see the following sections:

    For example, you could clone all of the PubSub+ Insights dashboards and then add filters to the widgets on the dashboards to reflect the needs of your team.

  • Create custom monitors and clone existing PubSub+ Insights monitors to customize them. For more information, see Creating Custom Monitors
  • Mute monitors whose default thresholds don't meet your needs, or to limit the amount of notifications you receive from specific monitor types, see Muting PubSub+ Insights Monitors for Datadog.
  • Schedule downtime for monitors. Scheduling downtimes allows you to turn off notifications from specific or all monitors for a specified time, once or on a recurring schedule. Scheduled downtimes are useful for managing notifications during maintenance, or upgrades. See Scheduling Downtime.

You must understand how to perform configurations in Datadog to create custom views, dashboards, and monitors. Solace professional services can provide help creating custom dashboards, views, filters, and monitors, getting you the results you want faster.

Cloning Existing PubSub+ Insights Dashboards for Datadog for Reuse

Your business may have its own unique requirements, but you might find that a small modification to one of the PubSub+ Insights dashboards for Datadog may help optimize its use. The PubSub+ Insights dashboards provided by Solace cannot be modified. If you need to modify a PubSub+ Insights dashboards to meet your organization's requirements, Solace recommends cloning one of the PubSub+ Insights dashboards and adapting it as required.

Before you clone PubSub+ Insights dashboards, Solace recommends becoming familiar with how to create and modify dashboards in Datadog. In addition to cloning, you can create your own dashboards and then add metrics and monitors to your dashboard. In either case, here are some suggested topics to help you get started with dashboards in the Datadog documentation:

After you clone a dashboard, you can do further customizations, including:

Creating Custom Monitors

Solace provides predefined PubSub+ Insights monitors for Datadog that you can use on your dashboards. You can create your own custom monitors or clone and customize existing monitors. You can base monitors you create on any metric or log that is already parsed as a monitor in your account. Cloning existing monitors accelerates the process.

Solace provides several sample template monitors that PubSub+ Insights dashboards can pick up. You can use the sample template monitors to create your own custom monitors. The sample template monitors have the solace_customizable_template tag to distinguish them from the other active monitors provided with Insights. You can find an example of cloning and customizing a monitor using a sample template monitor in Cloning and Customizing Template Monitors .

The template monitors contain instructions for cloning and configuring your own monitors. If you clone and configure other monitors, you must follow the guidelines listed below. Not following the guidelines can cause Solace to be unable to properly manage your event broker services and cause inconsistencies in the monitoring data reported in the dashboards.

  • You must have the Insights Advanced Editor or Insights Advanced Manager role to create, clone, and configure monitors.
  • You should clone only monitors that have been tagged with solace_insights.
  • Do not clone monitors tagged with solace_rmms. Incorrectly cloning solace_rmms tagged monitors can cause Solace to be unable to properly manage your event broker services, which may result in degradation of service. If you clone any monitors that clearly indicated to not be copied, it can result in you temporarily losing access to your Datadog account and all changes in your Datadog account being reset.
  • After you clone a monitor, remove the Solace-specific tags (solace-*), such as solace-insights and solace_customizable_template. Removing the Solace-specific tags prevents your cloned monitor from being modified or removed during upgrades or content refreshes.
  • If you want a custom monitor to be picked up in by the PubSub+ Insights dashboards, add the custom_insights tag to the monitor.
  • Consider the following if you clone an active monitor:
    • you get notifications from the original Solace-provided monitor as well as the cloned monitor that you created
    • if you add custom_insights to a cloned monitor, it can result in both monitors appearing in the PubSub+ Insights dashboards

For more information about monitors in general, see Manage Monitors in the Datadog documentation.

Cloning and Customizing Template Monitors

Solace provides several sample template monitors that you can clone and customize to create your own custom monitors that log metrics that are of importance to your organization. With the template monitors you can define the event or system logs to monitor, the alert conditions, who is notified when those conditions are met, and what message they receive.

The template monitors are provided as is to enable you to create your own monitors. Solace does not offer or provide support for monitors created using the custom template monitor.

The following procedure creates a new monitor that notifies you when a client has connectivity issues.

  1. In Datadog, on the Monitors page, select the List tab and enter template in the search field to find the sample template monitors.

    Image displaying an example of the sample template monitor

  2. Select a sample template you want to clone. In this example, select [Insights] Sample template for broker event log monitors.
  3. In the title bar at the top of the screen, select Clone.

    Screenshot showing an example as described by the surrounding text.

    The cloned monitor opens in edit mode indicated by the breadcrumb ending with Configuration in the top left corner. The name of the monitor is prepended with Cloning. You can now define your monitor's new parameters.

  4. In section 1, define the Monitor scope. Select Query and use the Define the search query fields to define the query. For example, enter @event:SYSTEM_CLIENT_CONNECT_FAIL as your search query.

  5. Next define the criteria that triggers the monitor using the fields below the search query field. Click in the field to the right of by, and enter service_name(service_name) and client_username(@client_username). You may need to click the add icon to add an extra field. This triggers a notification whenever this combination appears in the selected logs.

    Image displaying the Defint the search query section

  6. In section 2, set the conditions that cause an alert to trigger: Use the fields in this section to set the thresholds that trigger alerts and warnings. Note that the default values for both the Alert and Warning thresholds are set to unrealistically high values to prevent the monitor from triggering, preventing Datadog from sending you unnecessary alerts. You should define different thresholds for each which are meaningful for your organization. When defining your thresholds, Solace recommends using 1 as a starting point.

    Image displaying the options in the Set alert conditions section

  7. In section 3, Configure notifications & automations you must:

    • complete the notification email that is sent, and who it is sent to

    • remove the following tags in metadata section:

      • solace_insights
      • solace_customizable_templates
      • version:*.*.*

    • add the custom_insights tag in the metadata section to prevent your custom monitor from being affected by future updates

    You can optionally configure when renotifications are sent, additional metadata to include with the notification, including priority and tags, and configure aggregation.

    For more information, see Notifications in the Datadog documentation.

  8. (Optional) In section 4, click Edit Access if you want to restrict who can view and edit the monitor. For more information, see Monitor Permissions in the Datadog documentation.

  9. Click Create.

You have successfully configured a monitor that alerts you whenever a client has issues connecting to your services.

Muting PubSub+ Insights Monitors for Datadog

You can mute PubSub+ Insights monitors for Datadog. When you mute PubSub+ Insights monitors, they continue to trigger on alerts and warnings, without sending out notifications. You may want to mute PubSub+ Insights monitors so you can be notified at different thresholds than those set by default. You can clone one of the PubSub+ Insights monitors, set a different threshold, and then mute the original monitor, receiving notifications from your customized monitor only.

Muting PubSub+ Insights monitors does not affect the ability of Solace to monitor the operation of your services. However, practice caution when muting PubSub+ Insights monitors to avoid missing notifications that are relevant to your monitoring needs. Solace provides some proprietary monitors that may not be mutable without having the correct roles and permissions. You can mute these monitors by scheduling downtime using their monitor tags. See Muting Locked PubSub+ Insights Monitors for Datadog Using Downtime for more details.

  1. In Datadog, go to Monitors > Monitor List to access the list of monitors. You may need to toggle the False setting flag under the Active filter to the left of the table to see a full list of available downtimes.
  2. Select the monitor you want from the list.
  3. Click on Mute Mute button in the top title bar to open the Create a downtime to mute this monitor dialog.

    Image displaying the downtime dialog in Datadog

  4. Use the dialog box to set the duration, scope, and an optional comment for the downtime, and then click Create Downtime. For more information, see Monitor Status Page in the Datadog documentation.

Quickly Mute and Unmute Monitors

You can quickly mute and unmute PubSub+ Insights monitors for Datadog directly from the list of Monitors using the quick mute icon quick mute icon.

Image displaying the location of the Quick Mute button

Muting a monitor this way does so for an indefinite time period. You generally cannot mute proprietary [Insights] monitors this way, see Muting Locked PubSub+ Insights Monitors for Datadog Using Downtime for details.

Muting Locked PubSub+ Insights Monitors for Datadog Using Downtime

You may encounter some PubSub+ Insights monitors that appear to be locked to users with the Insights Advanced Manager or Insights Advanced Editor role. These locked PubSub+ Insights monitors are generally proprietary monitors provided by Solace.

You can see if a monitor is locked by selecting it from the monitor list, clicking View Configuration, and scrolling to the Define permissions and audit notifications section of the monitor. A locked monitor displays a statement similar to 'Restricted. Access is restricted to some.'

Image displaying an example of how a locked monitor appears in the Define permission and audit notifications section

If the monitor is locked, it can still be muted by scheduling a downtime using its tags, see Scheduling Downtime.

Scheduling Downtime

You can schedule downtime for PubSub+ Insights monitors in Datadog. A scheduled downtime allows you to mute PubSub+ Insights monitors starting at a scheduled start time, for a specific duration. Scheduled downtimes are useful when you plan upgrades or other work that may trigger notifications from the various PubSub+ Insights monitors. Perhaps you have a datacenter that has a maintenance scheduled on a specific weekday every week, causing notifications to be sent whenever the maintenance occurs. You can schedule a downtime for that period, muting notifications for that time. Once scheduled downtimes have been created, you can later edit, clone, or cancel them. For more information, see Edit, Clone, or Cancel a Scheduled Downtime.

Scheduling downtime does not affect the ability of Solace to monitor your services.

  1. In Datadog, go to Monitors > Downtimes to access the list of currently scheduled downtimes.

    Image showing location of Schedule Downtime button on Manage Downtimes tab of Monitor screen in Datadog

  2. Click schedule downtime icon to open the Schedule Downtime window.
  3. In step 1, choose the monitor to silence. You can select to schedule downtime for monitors by their monitor name or monitor tags using the Example of the By Monitor Name, By Monitor Tags toggle button toggle:
    • Selecting By Monitor Name displays the Monitor field where monitors can be selected from a list, or entered if the monitor name is known.
    • Selecting By Monitor Tags displays the Monitor Tag field where tags can be selected from a list, or entered if the tag name is known. Tags are attached to monitors and can be viewed at the top of any open monitor, above the monitor's name.

      Image displaying the location of TAGS for a selected monitor

    • You can also mute by scope such as service_id:some_id or queue_name:some_queue_name using the Group Scope fields.
    • Monitors that will be scheduled for downtime are shown in the panel on the right of the Schedule Downtime page.
  4. In step 2, configure the schedule for the downtime. Downtimes can be one time events, or reoccurring. You can also specify the day, start time, and end time for the downtime.
  5. In step 3, define a message that is sent to notify people about the scheduled downtime when it occurs. You can enter emails of the people to notify using @handle notation or select them from the menu in step 4.

Edit, Clone, or Cancel a Scheduled Downtime

Once you have created scheduled downtimes, you can edit, clone, or cancel them as required.

  1. In Datadog, go to Monitors > Downtimes to access the list of currently scheduled downtimes.
  2. Hover the cursor over the schedule you want to change. A menu appears from the right side of the monitor.

    Image displaying the location of the Edit, Clone, and Cancel menu on the Monitors list

  3. Click the icon related to the task you want to do:
    • edit icon - Edit - Opens the Edit Downtime screen. This screen is identical to the Schedule Downtime screen detailed above (Scheduling Downtime) but with the fields already populated. You can change the fields as needed then click Save. You are returned to the list of scheduled downtimes with your edited downtime open.
    • clone icon - Clone - Opens the Clone Downtime screen. This screen is identical to the Schedule Downtime screen detailed above (Scheduling Downtime) but with the fields already populated. You can change the fields as needed and click Clone. You are returned to the list of scheduled downtimes with your cloned downtime open.
    • Cancel downtime icon - Cancel - Opens a confirmation window. Click Confirm to remove the scheduled downtime from the list.

Creating Custom Views

The default view for a dashboard shows everything in a predefined time interval without any filters applied. You can create different views to give you different contexts for the information on your dashboard, which can be useful for troubleshooting or analyzing your estate.

When you apply data or time filters to a view, that filtering is not saved. If you want to save the view based on your filtering choices, perform these steps:

  1. After applying filters in the Filter by fields, click the Saved Views drop-down list.

  2. Select Save selections as view to open the Save View dialog.

  3. Enter a name and an optional description, then click Save.

    The saved view is now accessible from the Saved Views drop-down list.

For more information about managing custom views, see Saved Views in the Datadog documentation.

Creating Filters

In addition to the some of the predefined filters that are provided on the PubSub+ Insights dashboards, you can also create your own filters using a template variable to create additional filters. Template variables can give you different views of the data, helping you better understand different aspects of your estate.

To create a template variable, perform these steps:

  1. On the dashboard, click the Edit next to the existing Filter by fields.
  2. Click the add icon to open the Variable type dialog.
  3. You can add both Filter and Group by variable types:

    • Filter: Add a tag or attribute to filter dashboard queries and visualizations.

    • Group by: Add a tag or attribute to show a breakdown of groups in your data.

    4. Group by is only supported on select widgets.

  4. Enter a name for the variable in the Variable name field.
  5. Click Save.

For more information, see the Template Variables in the Datadog documentation.