Understanding Event Meshes and Mesh Manager

Modern enterprises now embrace event-driven architecture (EDA) as part of their digital transformation. EDA is software pattern where decoupled applications can asynchronously publish and subscribe to events; these events are managed by event brokers (in PubSub+ Cloud, these are event broker services). In a good EDA, you want decoupled applications to publish events to any event broker and are assured that subscribing applications receive those events in reliable and seamless fashion. As a result, every event-driven enterprise needs a central nervous system to quickly, reliably, and securely move events in real-time from where they are occurring to where they need to go. An event mesh provides this capability.

What's an event mesh

An event mesh is a configurable and dynamic infrastructure layer for distributing events among decoupled applications, cloud services, and devices. Mesh Manager configures an event mesh using event broker services.

An event mesh permits you to:

  • build a software architecture that ensures that messages (events/data) are securely and reliably received by subscribing applications using event broker services that are deployed and connected
  • share information between distributed, decoupled applications, which means that a publishing application can send events and another subscribing application to consume those events without knowing about each other
  •  to connect and easily scale hybrid - or multi-cloud event meshes that span multiple data centers; an event mesh ensures that events are exchanged between distributed, decoupled applications in real-time no matter where they are connected in a reliable manner

To help expand your understanding, let's consider this scenario to understand how an event mesh works. Imagine that you had multiple applications connected to event broker services in different geographic regions of the world that want to exchange events. Potentially, those applications are distributed in separate clouds or data centers. If those applications connect to the same event broker, events are exchanged seamlessly without the applications knowing about each other. However, those event broker services in this case aren't meshed together. This means that for a subscribing application to receive topics from publishers, it must be connected to the same event broker service.

To those, this consider this example where Subscriber A is connected to My-Second-Service (an event broker service), but it is interested in events from both Publisher D, and Publisher E.

  • Because the publishing applications are connected to different event broker services, Subscriber A only receives subscribed events from the publisher connected to My-Second-Service.
  • If Subscriber A wanted to receive events sent by Publisher E, it must be connected to My-First-Service as well.

As you can see in the following illustration below that though Subscriber A is interested in events (green boxes) from Publisher D and Publisher E (orange triangles). Subscriber A  receives the events from Subscriber D (indicated by the green check box), but the event from Publisher E never arrives to Subscriber A.

Animation that shows when event broker services aren't connected via an event mesh. Only events from Publisher D are received by Subscriber A, BUT events from Publisher E are not.

In comparison to the illustration below, if the applications were connected to event broker services that were configured as an event mesh, Subscriber A could connect to one event broker service, and can receive all subscribed events from the publishing applications that are connected to event broker services.

In this configuration, the event broker services are configured as an event mesh (green line), and you can see that subscribed events from both Publisher D and Publisher E are received by Subscriber A in a seamless manner.

Animation that shows when event broker services are connected via an event mesh. Events from both Publisher D and Publisher E are now received by Subscriber A.

As you can see in this simple scenario, an event mesh is useful and valuable for your event-driven architecture to ensure that distributed applications can receive events no matter where they are connected to receive the events they are interested in. The event mesh takes care of ensuring that the correct events are routed to the event broker services so that they are received by the subscribing applications.

For more information about how event meshes work PubSub+ Cloud, see Diving into an Event Mesh in PubSub+ Cloud and Considerations for Working with Event Meshes in PubSub+ Cloud.

When you're ready to start using Mesh Manager to create event meshes, see the following sections:

Diving into an Event Mesh

Architecturally, an event mesh is a logical layer that runs on your network to enhance its capabilities to allow distributed applications, microservices, and IoT devices to communicate via the publish/subscribe communication pattern. This allows for decoupled, dynamic communication in real-time.

At the core of an event mesh is DMR technology. Here's an overview of event meshes in PubSub+ Cloud, but for detailed information and DMR terminology, see DMR Overview and DMR Terminology:

  • To create an event mesh, using Mesh Manager, you can connect one DMR cluster to another DMR cluster. Mesh Manager creates a full mesh using Dynamic Message Routing (DMR). A full-mesh means that there are DMR external links that span between each DMR cluster in the event mesh. This permits for multi-site scaling. For more information, see Multi-Site Scaling.
  • A DMR cluster can contain one or more event broker services. By default, each event broker service you create in PubSub+ Cloud is its own DMR cluster, but optionally you can configure multiple event broker services to be part of the same DMR cluster by setting the cluster name. For more information, see Setting the Cluster Name.
  • Within a DMR cluster, each event broker service is referred to as a node. If you have multiple event broker services in a DMR cluster, the nodes are connected together via DMR internal links - which is useful for horizontal scaling and best used for event broker services in the same geographical regions. Currently, internal DMR links are configured with the help of Solace and aren't yet configured in Mesh Manager. For more information about horizontal scaling, see Horizontal Scaling.
  • In a DMR cluster, one event broker service (or node), is the gateway node that connects to other gateway nodes in separate DMR clusters. These connections are referred to as external DMR links, which is what Mesh Manager automatically configures for you.
  • External links take care of propagating events between DMR clusters, and internal links within the DMR cluster take care of propagating messages between the nodes. For more information, see DMR Subscription Propagation and Data Forwarding

After an event mesh is created, the event mesh takes care of routing events from producer applications (publishers)  to subscribing applications connected to other event broker services in each of the DMR clusters. Whenever a publishing application sends data or events to an event broker service, it routes the events/data based on the subscriptions to the consuming applications as long as the publishing and subscribing applications are connected to an event broker service on the same event mesh - in other words, they don't need to be the same event broker service. Furthermore, only events (or topics) that are subscribed to are sent to other nodes in the event mesh.

Just to solidify the understanding more, here's an example. In the following illustration, we have four event broker services, where each is in it's own DMR cluster - or a one node DMR cluster. For simplicity, we'll refer to the event broker service names. The single node in the cluster also acts as the gateway node to connect to other DMR clusters. In the example below, the event mesh is a mix of cloud providers including an on-premises Kubernetes deployment and configured a full-mesh.

The event mesh supports the following event- driven architecture:

  • Subscriber A is subscribed to events from Publisher C and Publisher E and connected to My-Second-Service.
    • Because Publisher C is connected to the same event broker service as Subscriber A, events are automated available.
    • Because Subscriber A is only connected to My-Second-Service, the event mesh ensures that events for (Publisher E are propagated from My-First-Service).
  • are sent to My-Second-Service from My-Fourth-Service. Because there are no subscriptions for events from Publisher D, no events from My-Fourth-Service are routed to My-Second-Service.
  • Subscriber B is subscribed to events from Publisher C, D, and E, but because none of those publishers are connected, the event mesh ensures that the events are sent from My-First-Service, My-Second-Service and My-Fourth-Service to My-Third-Service.

Note that if there was no event mesh (no green connections between the event broker services ), then Subscriber A could not get any subscribed events from Publisher C and Subscriber B would not receive subscribed events from Publisher C, E, or D because they are connected to different event broker services.

For geographically distributed use cases. Let's say you have event broker services in London, UK and New York, USA, an event mesh would be an easy way to bridge data between to ensure events that are published are received by subscribers no matter where they are connected. For example, when an application in London subscribes to events, that data would automatically start to flow across the event mesh (WAN); because data is only sent when it's being requested, or subscribed to, it very efficient from a bandwidth point of view. So no matter how many consumers in London are actually requesting that data, that data is sent only once.

You can interconnect on-premises and different cloud vendors which gives you a uniform data fabric without having to rely on cloud native services like Google Pub/Sub, or SQS or SNS and AWS.

This allows you flexibility to write applications that are agnostic to whether they are on-premises or off-premises.

Considerations for Working with Event Meshes in PubSub+ Cloud

The following are considerations when you create an event mesh using Mesh Manager in PubSub+ Cloud:

  • You require the following roles assigned to your user account based on the tasks you need to do:

    • To view and run Health Checks on an event mesh, your user profiles requires the Mesh Manager Viewer role

      To view, run Health Checks, create, modify, or delete an event mesh, you require the Mesh Manager Editor role

    • To create, modify, or delete event broker services as part of modifying or creating event meshes, you also require the Cluster Manager Editor role

    If you require a role assigned to you, contact your PubSub+ Cloud administrator. For more information about the roles, see Roles and Permissions.

  • Full-meshed DMR external links are created between the event broker service in your event mesh. If at anytime, there are less than two event broker service in an event mesh, the event mesh is deleted.

  • An event broker service can be part of only one event mesh. Anyevent broker services that are part of an event mesh must be Enterprise services; Developer services are not supported. If you are using DMR clusters, only one node in the DMR cluster can only belong to one mesh. For example, you cannot have one event broker service in the DMR cluster belong to one event mesh and a different event broker service in the DMR cluster belong to another event mesh. For best practices for configuring event meshes, see DMR Best Practices

  • Topic subscriptions are securely propagated only to event broker service that have subscribers (consumers) that are interested in the topic to minimize network traffic.

  • If you are connecting an event broker service that is deployed on a private network [Virtual Private Cloud/Virtual Private Network (VPC/VNet)] with an event broker service that is publicly accessible, the private deployment must initiate the connection.

    If both event broker services are deployed in a VPC/VNet or are publicly accessible, it doesn't matter which service initiates the connection. The VPC/VNet must both have IP connectivity  configured in the deployment. This can be done via VPN tunnels direct link, Transit Gateway, or Azure Spoke-Hub configurations.

  • In Mesh Manager, manually configured event meshes prior to Mesh Manager availability aren't seen. For example, event meshes that have been created using the PubSub+ Event Broker: Cloud (legacy mechanism). To see them, you must discover the event meshes. For more information, see Discover New Event Meshes.

Understanding Event Mesh Health

In Mesh Manager, you can validate the health of an event mesh using our Event Mesh Health Check (Health Check). The status of the event mesh is most useful for people that are administering connectivity between multiple event broker service located - usually located in different geographical regions. For a Health Check to run, the operational status of a link must be Up. The Health Check uses a customized, secure probe to check the integrity of an event mesh. The probe is an enhanced ping that performs an in-band validation and integrity test to verify that test events and messaging can be exchanged between event broker services; if the operational status of a link isn't up, the Health Check cannot run. A Health Check validates the following in an event mesh using one secure SMF connection on each event broker service:

  • Link status— Each direction for a link between two event broker services is pinged. This validates connectivity and the time that the probe is recorded.
  • Event status—We check the topics are properly propagated using the Request-Reply pattern to check that topics can be subscribed and published to using a reserved topic called #insights/pinger/ping. A temporary client username is created and removed after with the name mesh-validation-<session-id> where <session-id> is a string value representing the Health Check session.

An event mesh is considered unhealthy (or invalid) when at least one link is down or doesn't pass a Health Check. This can be more various reasons that include:

  • at the time, there were no Solace Message Format (SMF) connections available for the test to run
  • the tests between mesh nodes in either direction failed; the tests for both directions must pass for a link to be considered healthy

After a Health Check completes, information is available about the status when you view the details of the event mesh. For more information, see Viewing Information and Details of an Event Mesh.

For information about how to run a Health Check on an event mesh, see Running Event Mesh Health Checks.