Running Event Mesh Health Checks

An event mesh health check validates that events can be exchanged between the nodes in an event mesh. To run the health check on an event mesh, you require the Mission Control Viewer role. For more information about the role requirements, see Considerations for Working with Event Meshes.

For a health check to run, the operational status of all links must be Up. If the operational status is Down, you won't be able to run a health check and may need to troubleshoot the issue on the event broker service. For more information, see:

The health check uses an advanced pinger to check the connectivity between event broker services. A health check validates the following properties in an event mesh using one secure Solace Message Format (SMF) connection on each event broker service:

  • Link status—The health check pings each direction of a link between two event broker services. This process checks all links by sending a ping from each event broker service (or event broker) to the other event broker services in the event mesh. The health check records the time of the ping.
  • Event status—The health check uses a reserved #insights/pinger/ping topic to check topic propagation using the request-reply pattern to ensure that clients can subscribe to and publish to topics. The health check creates a temporary mesh-validation-<session-id> username, where <session-id> is a string value representing the health check session. The health check deletes the client username after the check completes.

Solace recommends running a health check if you make manual changes to an event broker service or are experiencing issues with your event mesh.

For more information, see:

Running Health Checks

To run a health check in Mesh Manager, perform these steps:

  1. Log in to the Solace Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the Solace Cloud Console.

  2. On the navigation bar, select Mesh Manager .

  3. On the Mesh Manager: Event Meshes page, click the tile for the event mesh on which you want to run a health check.

  4. Click View Health Check to open the Event Mesh Health Check dialog.

  5. Click Run Health Check.

  6. As the health check runs, the progress appears in the Event Mesh Health Check dialog. If you keep the dialog open, it shows the results for each test as it completes. You can expand the entry for each event broker service to see the details of the test.

    Screenshot showing the elements described in the surrounding text.

    If you click Close before the health check completes:

    • If you stay on the Event Mesh Details page, In Progress appears with a progress spinner under the Latest Health Check in the Event Mesh Details panel. The status changes to Success when the health check completes.

    • If you return to the Mesh Manager: Event Meshes page, the tile for the event mesh turns gray, and a progress bar appears at the bottom of the tile while the health check runs.

    • If you go to another part of the Cloud Console (for example Cluster Manager), the health check runs in the background, but it does not notify you when it completes. Return to Mesh Manager and select the event mesh to see the results.

    For information, see Viewing the Status of a Health Check and Links in an Event Mesh.

Handling Failed Health Checks

An event mesh is unhealthy if at least one link between any of the event broker services fails the health check test. It's important to note that the health check test reflects the health of the event mesh itself and not the individual services. To understand how to view the status of a health check, see Viewing the Status of a Health Check and Links in an Event Mesh.

You can identify the failed link and find useful information to help resolve the issue.

If the health check is not successful, some artifacts created for the health check may not be cleaned up as expected. After a health check, you may need to:

  • Delete the temporary mesh-validation-<session-id> username from each event broker service within your event mesh.

You must set certain properties in the ACL profile for your event broker service to Allow. If you have configured these properties to Disallow, the health check will fail. See Configuring ACL Profile Properties When Using the Event Mesh.

Troubleshooting Operational Links

You cannot run a health check if the operational status of any of its links is Down. To help in identifying the cause of the problem, use Broker Manager to identify and resolve the operational status of a link.

To open Broker Manager and troubleshoot the operational status of a link for an event broker service, perform these steps:

  1. Log in to the Solace Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the Solace Cloud Console.

  2. On the navigation bar, select Cluster Manager .

  3. Click the event broker service that you want to troubleshoot.

  4. In the top-right corner of the page, click Open Broker Manager. The Broker Manager web interface opens in another tab in your browser.

  5. In Broker Manager, click Clustering and then troubleshoot the links from there. For example, you could select the External Links tab to check if a link is down.

For more information, see Using Broker Manager with Event Broker Services.

Some common problems that may occur:

Linking to an event broker service that previously existed
If you deleted the service that was the second-to-last event broker service in an event mesh, links can remain, and you may need to manually remove the previous external links using Broker Manager.
One of the event broker services is in a Virtual Private Cloud/Virtual Network (VPC/VNet) or uses a private endpoint
If one of the event broker services has a public endpoint while the other has a private endpoint, the initiating event broker service must be the service with the private endpoint. To resolve this issue, switch the initiator so that the event broker service with the private endpoint, or the one connecting from a private region, is the initiator.
For more information, see Switch the Initiator for a Link on the Event Mesh.
Can't validate server certificates
If you are using custom TLS server certificates instead of the default Solace server certificates, you must upload those server certificates to each of the event broker services in your event mesh.
For more information, see Managing Custom TLS Server Certificates for an Event Broker Service.
Links between event broker services fail when both services are in different regions or both use private endpoints in different regions
The IP connectivity between private regions (for example, Customer-Controlled Clusters) is the responsibility of your organization. Verify the connectivity between regions to ensure event mesh creation is possible.

Configuring ACL Profile Properties When Using the Event Mesh

To successfully use Mesh Manager, you must set the following access control list (ACL) profile properties for event broker services in the event mesh to Allow:

  • Client Connect Default

  • Publish Default Action

  • Subscribe Default Action

Solace sets these properties to Allow when it generates the ACL profiles during service creation. If you set these properties to Disallow, the health checks you perform on your event mesh will fail. To configure the ACL profiles in Broker Manager on the Access Control tab, perform these steps:

  1. Log in to the Solace Cloud Console if you have not done so yet. The URL to access the Cloud Console differs based on your authentication scheme. For more information, see Logging In to the Solace Cloud Console.

  2. On the navigation bar, click Cluster Manager .

  3. On the Services page, select the card for the event broker service you want to configure and then click Open Broker Manager.

  4. In Broker Manager, select Access Control.

  5. On the Access Control page, click the ACL Profiles tab.

    You can review the configuration of each property in an ACL profile in the table. If you see the Client Connect Default, Publish Default Action, or Subscribe Default Action properties set to Disallow, you must change them to Allow.

    Screenshot showing the elements described in the surrounding text.

  6. Click the ACL profile you want to change. Note that you cannot change the properties for the #acl-profile profile.

  7. On the ACL Profiles page, click the tab for the property you want to change. For example, click Publish Topic to change the Publish Default Action property.

    Screenshot showing the elements described in the surrounding text.

  8. Click Edit.

  9. Click in the property field and select Allow and then click Apply.

  10. Repeat steps 7 through 9 until you have set all the required properties to Allow.

  11. Click Back to return to the ACL Profiles page.