Understanding Event Meshes and Mesh Manager
Modern enterprises now embrace event-driven architecture (EDA) as part of their digital transformation. EDA is software pattern where decoupled applications can asynchronously publish and subscribe to events; these events are managed by event broker services in PubSub+ Cloud. In a good EDA, you want decoupled applications to publish events to any event broker service and are assured that subscribing applications receive those events in reliable and seamless fashion. As a result, every event-driven enterprise needs a central-nervous system to quickly, reliably, and securely move events in real-time from where they are occurring to where they need to go. An event mesh provides this capability.
What's an event mesh?
An event mesh is a configurable and dynamic infrastructure layer for distributing events among decoupled applications, cloud services, and devices. Mesh Manager builds an event mesh by connecting event broker services.
An event mesh permits you to:
- build a software architecture that ensures that messages (events/data) are securely and reliably received by subscribing applications using event broker services that are deployed and connected
- share information between distributed, decoupled applications, which means that a publishing application can send events and another subscribing application to consume those events without knowing about each other
- to connect and easily scale hybrid - or multi-cloud event meshes that span multiple data centers; an event mesh ensures that events are exchanged between distributed, decoupled applications in real-time no matter where they are connected in a reliable manner
To help expand your understanding, let's consider this scenario to see how an event mesh works. Imagine that you had multiple applications connected to event broker services in different geographic regions of the world that want to exchange events. Potentially, those applications are distributed in separate clouds or data centers. If those applications connect to the same event broker service, events are exchanged seamlessly without the applications knowing about each other. However, those event broker services in this case aren't meshed together.
Consider this example where Subscriber A is connected to My-Second-Service (an event broker service), but it is interested in events from both Publisher D, and Publisher E.
- Because the publishing applications are connected to different event broker services, Subscriber A only receives subscribed events from the publisher connected to My-Second-Service.
- If Subscriber A wanted to receive events sent by Publisher E, it must be connected to My-First-Service as well.
As you can see in the following illustration below that though Subscriber A is interested in events (green boxes) from Publisher D and Publisher E (orange triangles). Subscriber A receives the events from Subscriber D (indicated by the green check box), but the event from Publisher E never arrives to Subscriber A. In this scenario, for a subscribing application to receive topics from a publisher, it must be connected to the same event broker service.
In comparison to the illustration below, if the applications were connected to event broker services that were configured as an event mesh, Subscriber A could connect to one event broker service, and can receive all subscribed events from the publishing applications that are connected to anyevent broker service. In this configuration, the event broker services are configured as an event mesh (line between the event broker services). As a result, subscribed events from both Publisher D and Publisher E are received by Subscriber A in a seamless manner.
As you can see in this simple scenario, an event mesh is useful and invaluable to ensure that distributed applications receive events no matter where they are connected. An event mesh takes care of ensuring that the correct events are routed to the event broker services so that they are received by the subscribing applications interested in those events.
For more information about how event meshes work PubSub+ Cloud, see Diving into an Event Mesh in PubSub+ Cloud and Considerations for Working with Event Meshes in PubSub+ Cloud.
When you're ready to start using Mesh Manager to create event meshes, see the following sections:
- Creating an Event Mesh
- Modifying an Event Mesh
- Running Event Mesh Health Checks
- Viewing Information and Details of an Event Mesh
Architecturally, an event mesh is a logical layer that runs on your network to enhance its capabilities to allow distributed applications, microservices, and IoT devices to communicate via the publish/subscribe (pub/sub) communication pattern. This allows for decoupled, dynamic communication in real-time.
At the core of an event mesh is DMR technology. To better understand how event meshes work in PubSub+ Cloud, you should familiarize yourself with DMR terminology and concepts. For more information about DMR, see DMR Overview and DMR Terminology:
- To create an event mesh, using Mesh Manager, you can connect one DMR cluster to another DMR cluster. Mesh Manager creates a full mesh using Dynamic Message Routing (DMR). A full-mesh means that there are DMR external links that span between each DMR cluster in the event mesh. This permits for multi-site scaling. For more information, see Multi-Site Scaling.
- A DMR cluster can contain one or more event broker services. By default, each event broker service you create in PubSub+ Cloud is its own DMR cluster, but optionally you can configure multiple event broker services to be part of the same DMR cluster by setting the cluster name. For more information, see Setting the Cluster Name.
- Within a DMR cluster, each event broker service is referred to as a node. If you have multiple event broker services in a DMR cluster, the nodes are connected together via DMR internal links - which is useful for horizontal scaling and best used for event broker services in the same geographical regions. These nodes are interconnected in a full mesh (where each node is connected all other nodes in the cluster). Currently, internal DMR links are configured with the help of Solace and aren't yet configured in Mesh Manager. For more information about horizontal scaling, see Horizontal Scaling.
- In a DMR cluster, one event broker service (or node), is the
gateway nodethat connects to other gateway nodes in other DMR clusters. These connections are referred to as external DMR links, which is what Mesh Manager automatically configures for you.
- External DMR links take care of propagating events between DMR clusters, while internal links within the DMR cluster take care of propagating messages between the nodes. For more information, see DMR Subscription Propagation and Data Forwarding.
After an event mesh is created, it takes care of routing events from publishing applications (publishers) to subscribing applications (consumers) connected to other event broker services in each of the DMR clusters. Whenever a publishing application sends events (or data) to an event broker service, the subscribing applications receive the events based on the matching subscriptions as long as the publishing and subscribing applications are connected to any event broker service on the same event mesh - in other words, they don't need to be the same event broker service. Furthermore, only events (or topics) that are subscribed to are sent to other nodes in the event mesh.
Just to solidify the understanding more, here's an example. In the following illustration, we have four event broker services, where each is in it's own DMR cluster - or a one node DMR cluster. For simplicity, we'll refer to the event broker service names. The single node in the cluster also acts as the gateway node to connect to other DMR clusters. In the example below, the event mesh is a mix of cloud providers including an on-premises Kubernetes deployment and configured a full-mesh.
The event mesh supports the following event- driven architecture:
- Subscriber A is subscribed to events from Publisher C and Publisher E and connected to My-Second-Service.
- Because Publisher C is connected to the same event broker service as Subscriber A, events are automated available.
- Because Subscriber A is only connected to My-Second-Service, the event mesh ensures that events for (Publisher E are propagated from My-First-Service).
- are sent to My-Second-Service from My-Fourth-Service. Because there are no subscriptions for events from Publisher D, no events from My-Fourth-Service are routed to My-Second-Service.
- Subscriber B is subscribed to events from Publisher C, D, and E, but because none of those publishers are connected, the event mesh ensures that the events are sent from My-First-Service, My-Second-Service and My-Fourth-Service to My-Third-Service.
Note that if there was no event mesh (no green connections between the event broker services ), then Subscriber A could not get any subscribed events from Publisher C and Subscriber B would not receive subscribed events from Publisher C, E, or D because they are connected to different event broker services.
For geographically distributed use cases. Let's say you have event broker services in London, UK and New York, USA, an event mesh would be an easy way to bridge data between to ensure events that are published are received by subscribers no matter where they are connected. For example, when an application in London subscribes to events, that data would automatically start to flow across the event mesh (WAN); because data is only sent when it's being requested, or subscribed to, it very efficient from a bandwidth point of view. So no matter how many consumers in London are actually requesting that data, that data is sent only once.
You can interconnect on-premises and different cloud vendors which gives you a uniform data fabric without having to rely on cloud native services like Google Pub/Sub, or SQS or SNS and AWS.
This allows you flexibility to write applications that are agnostic to whether they are on-premises or off-premises.
The following are considerations when you create an event mesh using Mesh Manager in PubSub+ Cloud:
You require the following roles assigned to your user account, depending on the tasks you need to do:
- To view and run Health Checks on an event mesh, your user profile requires the Mesh Manager Viewer role
- To view and run Health Checks, and to create, modify, or delete an event mesh, you require the Mesh Manager Editor role
- To create, modify, or delete event broker services as part of modifying or creating event meshes, you also require the Cluster Manager Editor role
If you require a role assigned to you, contact your PubSub+ Cloud administrator. For more information about the roles, see Roles and Permissions.
Mesh Manager always creates a full-mesh of DMR external links between the event broker service in your event mesh. If, at any time, there are fewer than two event broker service in an event mesh, the event mesh is deleted.
An event broker service can be part of only one event mesh. Any event broker services that are part of an event mesh must be Enterprise services; Developer services are not supported. If you are using DMR clusters, each node in the DMR cluster can belong to only one mesh. For example, you cannot have one event broker service in the DMR cluster belong to one event mesh and a different event broker service in the DMR cluster belong to another event mesh.
For best practices for configuring event meshes, see DMR Best Practices
Topic subscriptions are securely propagated only to event broker service that have subscribers (consumers) that are interested in the topic to minimize network traffic.
If you are connecting an event broker service that is deployed on a private network [Virtual Private Cloud/Virtual Private Network (VPC/VNet)] with an event broker service that is publicly accessible, the private deployment must initiate the connection.
If both event broker services are deployed in a VPC/VNet or are publicly accessible, it doesn't matter which service initiates the connection. The VPC/VNet must both have IP connectivity configured in the deployment. This can be done via VPN tunnels direct link, Transit Gateway, or Azure Spoke-Hub configurations.
In Mesh Manager, manually configured event meshes prior to Mesh Manager availability aren't seen. For example, event meshes that have been created using the PubSub+ Event Broker: Cloud (legacy mechanism). To see them, you must discover the event meshes. For more information, see Discover New Event Meshes.
In Mesh Manager, you can validate the health of an event mesh using the Event Mesh Health Check. The status of the event mesh is most useful for people who are administering connectivity between multiple event broker services, because these are often located in different geographical regions. For a Health Check to run, the operational status of a link must be
Up. The Health Check uses a customized, secure probe to check the integrity of the links in an event mesh. The probe is an enhanced ping that performs an in-band validation and integrity test to verify that test events and messaging can be exchanged between event broker services; if the operational status of a link isn't
up, the Health Check cannot run. A Health Check validates the following in an event mesh using one secure SMF connection on each event broker service:
- Link status—To validate connectivity, Health Check pings each direction of the link between two event broker services, and records the time of the probe.
- Event status—To verify that topics can be subscribed and published to, Health Check uses the Request-Reply pattern and a reserved topic called
#insights/pinger/ping. It creates (and later deletes) a temporary client username called
<session-id>is a string value representing the Health Check session.
An event mesh is considered unhealthy (or invalid) when at least one link is down or doesn't pass a Health Check. This can occur for a number of reasons that include:
- at the time, there were no Solace Message Format (SMF) connections available for the test to run
- the tests between mesh nodes in either direction failed; the tests for both directions must pass for a link to be considered healthy
After a Health Check completes, you can see the health status when you view the details of the event mesh. For more information, see Viewing Information and Details of an Event Mesh.
For information about how to run a Health Check on an event mesh, see Running Event Mesh Health Checks.