Topic Architecture Case Studies

The following case studies show how to apply topic architecture best practices:

• Case Study 1: Event-Driven Microservices, Choreography, and Event Sourcing
• Case Study 2: Event-Driven Integration with ASAPIO
• Case Study 3: IoT Devices—Command, Control, and Analytics
• Case Study 4: Data Ingestion and Distribution

Case Study 1: Event-Driven Microservices, Choreography, and Event Sourcing

Implementing microservices breaks monolithic applications (services) into smaller, more manageable components (microservices). Making microservices event-driven makes them more efficient, performant, scalable, and robust.

Two key attributes of event-driven microservices are:

• Choreography, where microservices react to changes in their environment, in contrast to the orchestration approach, where the system relies on either tight linkages between microservices or a central orchestrator that, although it drives microservice interactions, is the main bottleneck that prevents scaling.
• Event sourcing allows microservices to function and recover without needing atomic transactions for large units of work by capturing the system’s state within events rather than storing it in databases. You can achieve consistency in your system by replaying specific events, rather than managing the state of tightly-coupled services.

To learn more about using event-driven microservices, see:

• Event-Driven Microservices
• The Architect’s Guide to Event-Driven Microservices

To see an example of how to design a topic hierarchy for event-driven microservices, let's look at the "Sol-Beer" example in the Messaging Patterns for Event-Driven Microservices blog post.

As the blog explains, the removal of the point-to-point chaining of services has created a scenario very similar to the Middle Office example in the Data Ingestion and Distribution case study. Not only are all of the applications or services acting on the same event, but they also drive toward eventual consistency. The difference is that there could be follow-up events generated by services that gate, or otherwise affect other services. For example, the billing process may need to complete before the delivery vehicle can be dispatched. The following diagram shows the interrelationships between different topic domains, where some events can be processed independently but others are blocked. In our example, although you can't dispatch a delivery vehicle until after the billing clears, you can notify the delivery vehicles that there is an order being processed in their area.

Let's see how a well-defined topic architecture enables both the flow of events through this system and the flexibility to expand the data set. Originally, the system delivered beer by car, but as business grows, the company could add other products, such as Scotch whiskey, or add new delivery modes such as delivery from a drone, so they need to plan for business growth.

Building the Topic Root

In this system, customer-generated events are injected into the event mesh by the Sol-Beer WebApp. A customer places an order and then wants to be informed about updates to the status of their order, but they don't need details about logistics or inventory levels.

We can start from the event topic template, Domain/ObjectType/Verb/Version/Properties. First, focusing on the event topic root, our application domain is Sol-Beer’s online store, so store is appropriate as the Domain. One of the main functions of the store is to accept orders, so order may be an ObjectType. Within an order, a few actions are possible, including: created, updated, and canceled. Because this is the first version of the store, the version is v1.Based on these elements, we have the following root topics:

• store/order/created/v1
• store/order/updated/v1
• store/order/canceled/v1

Adding Topic Properties

After building a topic root, you can consider various properties for each individual Object/Action combination. Potential properties that may be useful in this application include:

Property	Description
{productGroup}	A value based on an enumeration of possible product group types (beer, scotch, and so on). This property may be useful for actions taken on specific product groups, such as marketing actions on a certain group of products.
{area}	The general region that an inventory manager is responsible for. This property may be useful for servicing multiple regions as delivery vehicles may only be interested in the orders in their region.
{productID}	The specific item that is ordered (dark beer, light beer, ten-year old scotch, and so on). This property may be useful for actions taken on specific products, for example a promotion on one certain product.
{customerID}	The customer identifier. This property may be useful for access control, because customers may only be able to place orders for their own ID.
{orderID}	The unique order identifier. This property may be useful to allow a customer to receive updates for a specific order.

Each new property is included when there is a routing, filtering, or access control use case. Each property should be ordered in the topic roughly by cardinality. In this example, there are more customers than there are products, and there are many more possible order IDs than there are customers. This leads to the following, complete topic:

store/order/created/v1/{productGroup}/{area}/{productID}/{customerID}/{orderID}

You can take a similar approach for the updated and canceled actions.

This topic taxonomy leads to many potential subscriber use cases. For example, the inventory management system may be set up to handle only specific regions. An inventory manager responsible for distribution in the Ottawa area would subscribe to the following topic:

store/order/created/v1/*/ottawa/>

Note the use of wildcards in the previous example indicates that the subscriber wants to receive messages with all possible values for that property. Here, the inventory manager is interested in orders for every product category (productCategory), so they use a wildcard (*) in the subscription to include all values.

Expanding the Event Catalog

In addition to processing orders, the Sol-Beer store processes payments and fulfills orders by local delivery. These events can be added to the catalog with similar topic roots as in the previous sample topics. For example:

• store/billing/paid/v1
• store/billing/nofunds/v1
• store/delivery/enroute/v1
• store/delivery/delivered/v1

If all events in the order, delivery, and billing process carry through the customerID and orderID, the process allows tracing and message replay for a single order or customer. This functionality is key for event sourcing. For example, when the billing events contain the customerID and orderID, you can trace the billing events back to order events. The resulting billing topics in the event catalog would be:

• store/billing/paid/v1/{productGroup}/{area}/{deliveryMode}/{productID}/{customerID}/{orderID}
• store/billing/nofunds/v1/{productGroup}/{area}/{deliveryMode}/{productID}/{customerID}/{orderID}

where the properties may look like:

beer/ottawa/drone/product456/cust4321/order12345

This rich topic taxonomy allows for many different use cases. For example, the inventory manager and the delivery vehicle that need to know when the billing has cleared for beer orders would subscribe to:

store/billing/enroute/v1/beer/ottawa/drone/product456/customer4321/order12345>

After the beer purchase has been authorized and the driver or drone assigned, the customer can track their delivery with live updates using delivery events. Some properties that may be helpful to include in the topic hierarchy are:

Property	Description
{vehicleType}	A value from an enumeration of vehicle type, such as a car, drone, and so on.
{latitude}	The latitude of the delivery vehicle position.
{longitude}	The longitude of the delivery vehicle position.
{vehicleID}	The delivery vehicle identifier.

The resulting event catalog entries would be:

• store/delivery/enroute/v1/{vehicleType}/{vehicleID}/{customerID}/{orderID}/{latitude}/{longitude}
• store/delivery/dispatched/v1/{vehicleType}/{vehicleID}/{customerID}/{orderID}

The event topic properties may look like this:

store/delivery/enroute/v1/car/vehicle4321/order12345/cust4321/45.421532/-75.697189

Some possible subscriptions for this event include:

• a customer tracking the delivery of all of their orders could subscribe to:

  store/delivery/enroute/v1/*/*/cust4321/>

• A fleet manager tracking all their vehicles, regardless of type or current order, within a geographical area might subscribe to:

  store/delivery/enroute/v1/*/*/*/*/45*/-75*

Flexible subscriptions allow events to be reused in new ways as necessary. For example, if new regulations required commercial drones to be tracked and reported to the city, a reporting agent could subscribe to the geographical area that it is responsible for:

store/delivery/enroute/v1/drone/*/*/*/45*/-75*

Case Study 2: Event-Driven Integration with ASAPIO

In this example, we'll event-enable SAP with ASAPIO . This is an example of event driven integration, where the data transformation capabilities of an iPaaS is combined with the real-time dynamic choreography of an event broker and event mesh. As shown in the diagram below, when a new purchase order is created in an SAP system, ASAPIO can push an event notification out to an event mesh in PubSub+, which streams that notification out to multiple interested applications (on-premises and in the cloud), such as applications used for billing, manufacturing and support. The value of this use case is that instead of each downstream application polling the system of record for changes, the system of record emits one event per change and all subscribing downstream applications have the ability to consume the event. This architecture reduces the workload on the system of record.

For more information about this SAP-specific system of record use case, see Solace with ASAPIO.

With ASAPIO, there can be different encodings (JSON, XML, etc.) of the same event. Because there can be only one event schema per event, we need to treat each encoding as a separate event. In this case, we include a HandlingInstruction property in the event root. We also move the HandlingInstruction property before the version property so that if just a single encoding changes, we can version just the one event. The result is the following event topic template:

Domain/ObjectType/Verb/HandlingInstruction/Version/Locality/SourceId/ObjectId

Let's see how a well-defined topic architecture helps data flow through this complex system.

In this use case, an event is emitted each time certain business objects (BO) change in the SAP system.

The events flowing from SAP to downstream applications have one of three different object types or aspects, each with progressively more information, which is controlled through ACLs. These aspects are described in the following table:

Aspect	Description
sap-bo-event	A notification that a particular business object has been created or updated. The topic and payload contain enough information for an application or user to retrieve the business object according to their SAP access rights.
sap-bo-metadata	Contains the information from sap-bo-event plus the IDs of the fields that have been updated. This allows an application to further filter by fields they are interested in before retrieving the business object.
sap-bo-data	Contains the information from sap-bo-event plus the actual fields and the changed values. This allows an authorized application to instantly get the changes with their values.

For these SAP events the topic template would be:

sap/{resourceType}/{action}/{representation}/{version}/{aspect}/{siteID}/{orgID}/{systemID}/{resourceID}

Item	Description
{sap}	A literal value specifying that these are SAP-originating events.
{resourceType}	The business object type that the notification applies to. For example, BUS2032 is a sales order.
{action}	A value from an enumeration of actions, for example create or update.
{representation}	The encoding of the event (e.g., XML, JSON).
{version}	The version of the event.
{aspect}	A value from an enumeration of object types sap-bo-event, sap-bo-metadata, or sap-bo-data.
{siteID}	The siteID, plantID, or locationID where this BO is owned, where GLOBAL indicates a global entity, and EMPTY indicates that there is no siteID associated with the notification.
{orgID}	The organization unit that owns the business object, where GLOBAL indicates a global entity, and EMPTY indicates that there is no orgID associated with the notification
{systemID}	The SAP instance or system ID.
{resourceID}	The business object ID.

The event catalog entries, which are derived from the event topic root, would look like:

• sap/BUS2032/create/JSON/v1
• sap/BUS2032/update/JSON/v1

To see how this event topic root is used, consider the following event:

sap/BUS2032/update/JSON/v1/sap-bo-data/empty/1000/ERD100/0000096852

As we read through the fields in the topic hierarchy, we see the following information:

• SAP event
• Business Object: BUS2032 (sales order)
• It's an update to a business object
• Encoded in JSON
• API version: 1
• Aspect: sap-bo-data (contains actual data)
• Site ID: none
• Organization ID: 1000
• SAP instance: ERD100
• Business Object ID: 96852

A downstream application that wants to receive all sales order updates with their data from organization 1000 could subscribe to:

sap/BUS2032/update/JSON/v1/sap-bo-data/*/1000/>

Note that this application would need the appropriate access permissions (via an administratively applied ACL) to allow it to subscribe to sap-bo-data events from organization 1000.

A downstream application that wants to receive all sales order updates (event only) from any SAP instance, any organization, at any site, could subscribe to:

sap/BUS2032/*/JSON/v1/sap-bo-event/>

Again, this application would need to have the appropriate access permissions to allow it to receive notification events of new sales orders.

Case Study 3: IoT Devices—Command, Control, and Analytics

Internet of Things (IoT) use cases traditionally involve data movement exclusively from field sensors into backend state capture or analytics systems. The initial goals created simple systems that could horizontally scale to handle millions of devices. IoT uses cases have evolved as the numbers and complexity of the devices have evolved. Security is now mandatory, and command and control of end devices is much more common within these systems. The events coming from IoT sensors are now counted on for more than simple statistics. They now also drive changes in upstream systems. Some critical sensor events might require human interactions, and it is important that the events are accurate and valid. For example, consider an air-bag deployment event on a mobile vehicle and the consequences if these events are spoofed or ignored. There are now also many use cases where IoT sensors receive data from backend systems, for example, connected vehicles receiving weather and traffic updates.

Modern industrial IoT use cases integrate a large number of components into a unified system. To effectively organize and route data across these systems, a unified namespace (UNS) is needed. A UNS is a single source of truth for the real-time state of an enterprise. It allows all components within the enterprise to communicate bidirectionally in a standard way. An event mesh provides this functionality, and a well-defined topic hierarchy is key to standardizing communication, particularly by making data identifiable across all components in the system. For instance, an event published from an IoT device may be consumed by a number of backend systems, from data lakes to analytics engines. Event topics must be well constructed to be useful in these backend systems and prevent data lakes from turning into data swamps. In particular, a device ID should be included in the topic properties so that all events for a particular device can be monitored in real time through the event mesh and viewed historically in backend systems. Including the device ID in topics also allows you to secure the data through ACL substitution expressions.

To learn more about the connected car use case, see the Solace IoT Event Mesh article.

Let's see how a well-defined topic architecture helps authorize and route events from remote device to backend systems, and how backend systems can control remote devices.

Device to Backend Services

Events emitted by edge IoT devices consist mainly of sensor data.

Starting with the topic event template of Domain/ObjectType/Verb/Version/Properties, we have a topic template of

fromVehicle/{vehicleComponent}/{action}/{version}/{severity}/{username}/{sensorID}

Using fromVehicle as the domain indicates that the information in the topic is coming from the sensors. The domain may need to include additional information if multiple vehicle generations are hosted on the same event mesh, for example, myCar/myGeneration/fromVehicle.

Property	Description
fromVehicle	A literal value specifying that these are events from the vehicle.
{vehicleComponent}	The name of the controllers or apps within the vehicle (e.g.,TCU and MCU).
{action}	A value from an enumeration of sensor actions (e.g., reset, update, and failure). Different vehicle components may have different actions.
{version}	The version number for this release of fromVehicle.
{username}	A value from an enumeration of severity (e.g., error, warning, and info).
{sensorID}	The unique sensor that produced the event.

The event catalog entries derived from the event topic root, would look like:

• fromVehicle/tcu/reset/v1
• fromVehicle/tcu/update/v1
• fromVehicle/tcu/failure/v1
• fromVehicle/mcu/reset/v1
• fromVehicle/mcu/update/v1

The properties of the topics provide more metadata for routing and filtering. For example, the location of the vehicle might affect event routing and the severity of the event might affect the number of applications interested in the event. Each topic should also include the sensor ID because it is crucial for access control restrictions to ensure that data from other sensors can't be spoofed.

Let's say a vehicle has an accident and its airbag deploys. The vehicle might emit an event with topic:

fromVehicle/tcu/update/v1/critical/vin4321/airbag

There could be an application listening to all critical severity events:

fromVehicle/*/update/v1/critical/>

In many cases, IoT devices are not in a secure location and can be subject to physical or software hacking. For this reason, the device must be authenticated on every connection, and the authenticated ID must be included and validated with each message to ensure the device only sends authorized events. This is the reason the MQTT username field is so important in this type of environment—it prevents one IoT device from pretending to be another IoT device.

For example, verifying the username would prevent a hacked vehicle with username vin4321 from sending an event as vin1234. The following update from username vin4321 would be allowed:

fromVehicle/tcu/update/v1/critical/vin4321/airbag

However, the following update from username vin4321 would not be allowed:

fromVehicle/tcu/update/v1/critical/vin1234/airbag

Backend Services to Device

The data sent from backend applications to vehicles could consist of multicast messages to several vehicles or unicast messages to a specific vehicle. Multicast messages could be based on location, on-board application, vehicle type, or other criteria. For unicast messages, the username ensures that only the intended vehicle receives the update.

Consider a topic template of:

toVehicle/{vehicleComponent}/{action}/{version}/{location}/{clientID}/{sensorID}

Field	Description
toVehicle	A literal value specifying that these are events sent to the vehicle.
{vehicleComponent}	A value from an enumeration of the controllers or apps, for example TCU, MCU, and so on. An MCU could act as multiplexor to downstream subcomponents.
{action}	A value from an enumeration of actions (possibly dependent on the vehicle component).
{version}	The version number for this release of orders.
{location}	The Multicast area, group, or null.
{username}	The username or device identifier used to validate whether the sender is authorized to produce the event.
{endpointID}	The identifier of the specific sub-component or app that should receive the control message.

The event catalog entries, which are derived from the event topic root, would look like:

• toVehicle/tcu/update/v1

• toVehicle/mcu/update/v1

To differentiate multicast events that are available to all vehicles, an allVehicles application domain could be used:

• allVehicles/weather/update/v1

• allVehicles/traffic/update/v1

The properties of the topics would provide more information for routing, filtering, and access control.

A vehicle would subscribe to all events directed specifically to it:

toVehicle/*/*/v1/*/vin4321/>

The vehicle could also subscribe to weather updates in its region:

allVehicles/weather/update/v1/ottawa/weatherPlan4321/>

And it could subscribe to traffic updates for the current road and direction:

allVehicles/traffic/update/v1/417east/trafficPlan4321/>

As previously mentioned, the username is used to access policies and filtering. For example, the vehicle may only be allowed to subscribe to toVehicle domain updates where its VIN is included in the topic, preventing it from accessing information for other vehicles, whereas it may be allowed to consume any allVehicle updates. For a detailed discussion about access control in IoT applications, see Improving IoT Security with Access Control List Substitution Variables.

Case Study 4: Data Ingestion and Distribution

In this example, we look at data sources throughout an enterprise that write data into distributed data stores. These data sources could be published transaction logs from a database, online purchases, market data orders, and so on.

The following diagram illustrates an example of market data orders and post trade data, where the trade itself takes place in the Front Office and the post-trade activity takes place in the Middle Office. Here we can see that the client gateways produce events for the Front Office and the trading platform is the producer of events that flow to the Middle Office systems. We can also see the Middle Office is not a single-purpose consumer application, but a series of applications that each have specific tasks and data requirements.

For additional details about this use case, see Investment Banking and the Middle Office Shock Absorber.

Without a properly designed event topic architecture that allows event re-use, you might have a very expensive trading platform publishing the same event multiple times in multiple formats to satisfy each Middle Office need.

We start with the standard topic event template of Domain/ObjectType/Verb/Version/Properties.

Let's see how a well-defined topic architecture helps data flow throughout this complex system.

Front Office

The message exchange pattern in this phase of the interaction consists of a simple command message. As shown in the diagram above, the data flow for the Front Office system, works like this:

1. A Client registers an intent to buy or sell a product with a Client Gateway.
2. The Client Gateway turns this request into a Client Order and sends it to a Trading Platform system.
3. The Trading Platform system determines the appropriate Exchange to use, and places the Market Order.
4. The Exchange fulfills the Market Order and sends confirmation back to the Trading Platform system.
5. On receipt of a Fill event from the Exchange, the Trading Platform notifies the Client (via the Client Gateway) that its order has been filled.

For the Front Office, the topic template for client orders would be:

orders/{productGroup}/{action}/{version}/{market}/{source}/{product}

Item	Description
orders	The literal value specifying the topic describes the total of all orders events.
{productGroup}	A value from an enumeration of possible product groups.
{action}	A value from an enumeration of order actions (e.g., new, update, and cancel).
{version}	The version number for this release of orders
{market}	General region or group of markets for which the trading platform is responsible.
{source}	The customer identifier.
{product}	The specific item being ordered.

The event catalog entries, which are derived from the event topic root, would look like:

• orders/equity/new/v1
• orders/equity/update/v1
• orders/equity/cancel/v1
• orders/bond/new/v1
• orders/bond/update/v1
• orders/bond/cancel/v1

The event topic properties might look like:

orders/bond/cancel/v1/NA/pri1/customer1/appl

Monitoring apps could track individual customers using the following subscription:

orders/equity/*/v1/NA/custID4321/>

Actions like cancellations could be monitored to flag unusual activity using the following subscription:

orders/equity/cancel/v1/NA/>

Horizontal Scaling

A single application may not be able to keep up with large amounts of data. Horizontal scaling of consumers can be accomplished in many ways, such as nonexclusive queues or partitioned queues (see Message Distribution with Partitioned Queues. However, if the load is too much for a single event broker to handle, the data may need to be sharded across multiple event brokers in the event mesh using topic partitions.

In this use case, the properties of the topics provide more information for routing and filtering. The Smart Order Routers (SORs) could be sharded based on these properties and each process subsets of the products. This structure allows for horizontal scaling across multiple event brokers within the event mesh.

In this example, the SORs split the traffic, first by productGroup and then at lower levels to handle shards or the product space.

The first SOR instance could have the following subscriptions:

orders/equity/*/v1/NA/*/a*/>

...

orders/equity/*/v1/NA/*/m*/>

And the second instance could have the following subscriptions:

orders/equity/*/v1/NA/*/n*/>

...

orders/equity/*/v1/NA/*/z*/>

Using Subscription Exceptions for Horizontal Scaling

Consider an expected high-volume event, such as an IPO for ACME Rideshare (ACME), where a large volume of trades on one specific symbol is expected for a limited period of time. To handle this situation, you could deploy a dedicated event broker to handle only the ACME Rideshare trade events, with an SOR instance subscribing to orders/equity/*/v1/NA/*/acme/>.

To ensure that no ACME Rideshare events are propagated to other SOR instances, the SOR instance that handles the equities starting with "a" could add a negative subscription of !/equity/*/v1/NA/*/acme/>, allowing for dedicated handling of ACME orders by the other SOR instance.

Middle Office

After the Trading Platform system has completed a trade, it sends an event message into the Middle Office system. The Middle Office system includes several systems that must have a consistent view of the trades that happen, but where each needs specific data to perform its tasks. Because this is a separate domain, the SORs would publish to a topic under the transaction domain. The event includes the specific Exchange the trade should occur on, for example:

transaction/equity/new/v1/NYSE/traderID1234/Order4321

The Middle Office exchange gateways would subscribe to the exchanges and pools they service, for example:

transaction/equity/new/v1/NYSE/>

The full topic taxonomy of the Middle Office system would be:

transaction/{productGroup}/{action}/{version}/{venue}/{source}/{orderID}

Item	Description
transaction	The literal value specifying the topic describes the total of all orders events.
{productGroup}	A value from an enumeration of possible product group types.
{action}	A value from an enumeration of transaction actions such as new, updated, filled, canceled.
{version}	The version number for this release of transaction.
{venue}	A value from an enumeration of the liquidity pools or exchanges where transactions can occur.
{source}	The customer identifier.
{orderID}	The identifier for the unique product order.

In this example, you could imagine that applications like Settlements and Risk systems would be interested only in successfully completed transactions, whereas a Reporting system may require access to every transaction including canceled ones, and Trade Data Capture system might consume both every order event and every transaction event, for either the entire set of data or again sharded by product type, region, venue, or product.

For example, for the event:

transaction/equity/new/v1/NYSE/traderID1234/Order4321

The Settlements system may subscribe to:

transaction/equity/filled/v1/>

or:

transaction/equity/filled/v1/NYSE/>

While the Risk system might subscribe to the following topic to monitor the positions of a particular trader:

transaction/equity/*/v1/*/traderID1234/>