Monitoring Your DMR Network

If there are problems with your event mesh, for example if Dynamic Message Routing (DMR) is configured incorrectly or if there are operational problems in the network, one or more event brokers may report topology errors or DMR being operationally down.

For information about verifying the health of a DMR network created with Mesh Manager in the Cloud Console, see Verifying the Health of an Event Mesh.

Topology Errors and Troubleshooting

Topology errors are caused by problems in the configured or operational network topology.

Topology problems are expected while adding and removing nodes in a network, because configuration changes need time to propagate through the event mesh. However, if you follow the procedures in this documentation, no message loss is expected, and the topology problem should be temporary and clear by itself.

Some examples of topology errors are:

  • missing links — every node in a cluster must be connected by an internal link. Links must be bi-directional; that is, each link must be configured correctly on both nodes.

  • mismatched link span — a link is either internal (connecting two nodes within a cluster) or external (connecting gateway nodes in two different clusters). The configuration at both ends of the link must match.

  • missing channels — each link must have a control channel plus one data channel per Message VPN.

  • node name mismatch — each node must correctly specify the name of the remote node connected by the link.

  • DMR is not correctly enabled — each participating VPN must opt in to DMR. In addition, for nodes connected by an internal link, the list of Message VPNs participating in DMR on each node must be the same.

DMR can be operationally down for other reasons, such as:

  • misconfiguration in other areas, such as replication or redundancy

  • problems with Guaranteed Messaging (for example, the message spool is down)

  • issues with cluster authentication

  • issues with cluster synchronization

  • a dangling link — a cluster link that is configured but not operational, typically because the node at the other end was removed, was never fully configured, or cannot be reached. A dangling link does not clear itself. It stays configured and Down until you correct or remove it.

    Dangling links are easy to overlook, but they can block service. A restarting event broker waits for subscription state from the nodes it expects to hear from. If it depends on a node that is unreachable and no other source can supply that state, service for the affected Message VPNs stays blocked. For more information, see Troubleshooting Cluster Synchronization Issues.

If there is a topology error, or DMR is down for some other reason, subscription propagation and data forwarding throughout the network may not be working properly.

To troubleshoot issues with your event mesh, use the CLI commands in the sections that follow to view the configuration and operational status of the objects in the mesh. Start by looking at the cluster information, then the links, then the channels. That is, start at the widest level of detail (the cluster) and narrow down your focus (first link, then channel) as needed.

For information about using the CLI, see Accessing the CLI for Event Broker Services and Logging In to the CLI (for appliance event brokers and software event brokers).

Finding and Fixing Link Problems

To find link problems, enter show cluster * link * and look for any link whose operational state is not Up. Also confirm that each link is configured on both of the nodes it connects.

To resolve a link problem, take one of these actions:

  • Correct the cause (for example, network connectivity, the link configuration on either node, or the remote node name) so that the link becomes Up.

  • If the link is no longer needed, remove it from both of the nodes it connects, so that neither node keeps a configuration that points at the other.

Displaying Cluster Information

To display cluster information, enter the following command:

show cluster <cluster-name-pattern> [detail]

Where:

<cluster-name-pattern> displays clusters matching the pattern. The cluster pattern can be the cluster's full name, or part of its name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name. Entering only the wildcard character * for the name displays all clusters.

detail displays detailed information for the specified cluster.

Displaying Cluster Synchronization Information

When you enter the show cluster command, in addition to other information about the cluster, the event broker also displays whether cluster synchronization (cluster-sync) has completed.

Cluster synchronization is the mechanism by which event brokers that have just restarted, and so have no knowledge of remote event broker subscription needs, learn of those subscriptions before providing local service.

In cases where cluster synchronization cannot be achieved, service (on a per-Message VPN basis) is blocked until the issue causing the local event broker to be unable to learn of its remote subscriptions is corrected. Cluster synchronization has no timeout. Being unable to achieve cluster synchronization in a timely manner is generally an operational or configuration issue with another event broker in the network.

Troubleshooting Cluster Synchronization Issues

This section helps you diagnose and resolve issues when your event broker fails to achieve cluster synchronization after a restart. Cluster synchronization problems typically manifest as event brokers that remain in a synchronizing state longer than expected.

After a restart, an event broker relearns its own past subscription set for its router name from its cluster neighbors and its high availability (HA) mate. Every event broker in a cluster learns the subscriptions of all the others, so there are usually several sources for the same subscription state. For example, if event broker A needs the subscriptions of event broker B, and B is down, A may still get them from a third event broker C. An event broker's HA mate is another independent source. The inactive event broker in an HA pair can get its subscription state only from its active HA mate. Cluster synchronization fails outright only when no reachable source can provide the needed state.

Causes of Cluster Synchronization Failures

Cluster synchronization is primarily blocked by an event broker's inability to communicate with another event broker that can supply the subscription state it needs. Common causes include:

  • DMR is not operationally Up (with no topology errors) before the restart. A link that was already degraded before the restart makes it more likely that sync will block afterward.

  • Network connectivity issues preventing the event broker from reaching one or more cluster peers, or peer event brokers that are unreachable or not operational and hold needed subscription state, with no alternative source (another cluster event broker or an HA mate) able to provide it.

  • DMR misconfiguration, such as a one-way cluster link, DMR enabled on only one of two event brokers for a given Message VPN, or missing or misconfigured gateway nodes or cluster links.

  • Missing or misconfigured Message VPN settings, such as mismatched DMR bridge configuration on external cluster links, or an incorrect remote Message VPN.

Consequences When Cluster Sync Fails

These consequences are the same regardless of event broker type (software, appliance, or event broker service):

  • Service is blocked on a per-Message-VPN basis. The affected Message VPNs have no local service until synchronization completes. A normal synchronization completes promptly, so a long delay indicates a problem.

  • Cluster synchronization has no timeout. The event broker remains in a synchronizing (blocked) state indefinitely; it does not resume automatically. An event broker stuck synchronizing is generally a signal of an operational or configuration problem elsewhere in the network, not a transient condition to wait out.

Troubleshooting Steps

If your event broker remains in a synchronizing state longer than expected, perform these steps to troubleshoot the issue:

  1. Verify network connectivity between all event brokers in the cluster, identify any unreachable event brokers, and confirm that all event brokers in the cluster are operational.

  2. Check the DMR configuration. There may be a misconfiguration on either the local or the unreachable event broker (for example, a one-way cluster link may be configured). To display detailed information about your clusters, enter show cluster * detail. For more information, see Displaying Cluster Information.

  3. Verify that all Message VPNs are properly configured on both the local and unreachable event broker (for example, DMR may be enabled on only one of the two event brokers). To display DMR information for a Message VPN, enter show message-vpn <vpn-name> dynamic-message-routing. For more information, see Displaying Message VPN Information.

  4. On external cluster links, check for mismatched DMR bridge configuration settings for a particular Message VPN (configuration may be missing, or an incorrect remote Message VPN may be configured). To display the DMR bridges for a Message VPN, enter show message-vpn <vpn-name> dynamic-message-routing dmr-bridge <remote-node-name-pattern>.

  5. Check the cluster link status on neighboring event brokers. To display the status of each cluster link, enter show cluster * link *. For more information, see Displaying Cluster Link Information.

  6. Verify that all gateway event brokers are correctly configured across the cluster, and that each link you expect appears in the show cluster * link * output and is operationally Up.

When to Contact Support

If your event broker remains unable to complete cluster synchronization after you follow these troubleshooting steps, perform these steps:

  1. Run the gather-diagnostics command to collect system information.

  2. Note any error messages or unusual behavior.

  3. Document the steps you've already taken to resolve the issue.

  4. Contact Solace support with this information.

Displaying Cluster Link Information

To display cluster link information, enter the following command:

show cluster <cluster-name-pattern> link <link-name-pattern> [detail | client-profile | queue | ssl]

Where:

<cluster-name-pattern> displays clusters matching the pattern.

<link-name-pattern> displays cluster links matching the pattern. The link pattern can be the link's full name, or part of its name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name. Entering only the wildcard character * for the name displays all cluster links.

detail displays detailed information for the specified link.

client-profile displays cluster link client profile information for the specified link.

queue displays cluster link queue information for the specified link.

ssl displays cluster link TLS/SSL information for the specified link.

Displaying Cluster Channel Information

To display cluster link channel information, enter the following command:

show cluster <cluster-pattern> link <link-pattern> channel message-vpn <vpn-name> [detail]

Where:

<cluster-name-pattern> displays clusters matching the pattern.

<link-name-pattern> displays cluster links matching the pattern.

<vpn-name> displays channel information for the specified Message VPN.

detail displays detailed channel information for the specified link.

Displaying Message VPN Information

To display DMR information for a particular Message VPN, enter the following command:

show message-vpn <vpn-name> dynamic-message-routing [dmr-bridge <remote-node-name-pattern>]

Where:

<vpn-name> displays DMR information for the specified Message VPN.

<remote-node-name-pattern> displays information for DMR bridges that connect to the specified remote node.

Other Ways to Monitor DMR

In addition to the show commands, you can monitor DMR state in the following ways:

  • Syslog events — The event broker generates the SYSTEM_CLUSTERING_CLUSTER_DOWN, SYSTEM_CLUSTERING_LINK_DOWN, and SYSTEM_CLUSTERING_TOPOLOGY_PROBLEM events when a cluster goes down, a link goes down, or a topology problem is detected. Forward syslog to your monitoring platform and alert on these events. These events report state changes only. They do not report bandwidth or performance problems on a link that is up. For more information, see Monitoring Using Syslog and Solace Syslog Events.

  • SEMP — The Solace Element Management Protocol (SEMP) v2 monitor API returns the same cluster, link, and DMR bridge state as the show commands, in a structured format that you can poll from your own tools. The relevant resources are /dmrClusters, the links under each cluster, and /msgVpns/{vpn}/dmrBridges. For more information, see SEMP.

  • Insights — Solace Insights collects the dmr.cluster.status and dmr.link.status metrics for event broker services. For more information, see Solace Insights Metrics and Checks.

Checking DMR Health Before Restarts and Changes

Before you restart an event broker that is part of a DMR cluster, or make a change that restarts one (such as an upgrade or a topology change) or may cause cluster synchronization (such as an HA or replication failover), confirm that DMR is healthy. An event broker that restarts while DMR is unhealthy is likely to become blocked on cluster synchronization, and service for the affected Message VPNs does not resume until the problem is fixed. For more information, see Troubleshooting Cluster Synchronization Issues.

Before you proceed, confirm that these conditions are true:

  1. All cluster links are operationally Up. Enter show cluster * link * to check.
  2. No topology errors are reported. Enter show cluster * to check.
  3. Cluster synchronization is complete on every node. The show cluster output indicates whether cluster synchronization is complete.
  4. If the event broker is part of an HA pair, its HA mate is operational and DMR is operationally Up on the mate. The mate is an independent source of subscription state.
  5. If the event brokers are replication mates, the internal link between the mates is Up, and cluster synchronization is complete on both sites.
  6. No DMR-related alerts are active in your monitoring system.

If any of these checks fails, do not proceed. Resolve the problem first. After the restart, repeat the first three checks to confirm that cluster synchronization completes.