Topic Architecture Best Practices

Event-driven architecture and event-driven microservices have proven to be valuable application design patterns. The key advantage to both types of systems is that applications can collaborate by asynchronously reacting to event stimuli, instead of synchronously polling for events or relying on an orchestration layer. To get the correct event stimuli to the applications, there needs to be a way to asynchronously deliver events.

The publish-subscribe message exchange pattern is a common way to achieve this interaction. Beyond simple publish-subscribe messaging, other attributes are required to fulfill the intended design of an event-driven architecture. One of these attributes is a well-architected topic hierarchy that allows collaborative applications to efficiently attract and consume events.

In the sections that follow, we demystify the best practices for creating an event topic architecture by discussing:

Before designing a topic structure, you should have a good understanding of both topic syntax (for example, topic length, number of levels, wildcard characters, and so on) and the way topics control the flow of messages in the broker. Here are some resources:

For additional background, see the following:

Topic Architecture Definitions

The terms below are described as they relate to an event-driven architecture.

Event

An event can be broadly described as a change notification within your organization, system, or application domain. Events can have a variety of forms, but all have the common component of an action that has occurred on an object. For example, a command, which is a request for a data change, could have REST verbs, such as PUT, POST, and DELETE. Then the resulting notification event could contain a status that describes the change to the object, for example, UPDATED or DELETED.

Some more specific examples could be:

  • A data system: A command in this type of system could be a REST PUT, POST, or DELETE request. The request would invoke a data change in a database by triggering an embedded function call, which then reacts to the add, modify, or delete command, and emits a response at completion. The difference with event-driven messaging is that, unlike a database, the event mesh is an open and distributed system where the events can be distributed to all the interested applications. This allows for a real time distribution of data, because applications can register their interest and then asynchronously receive events instead of polling for changes or receiving batch updates.
  • An IoT system: An event here would represent a state change of an object, which models the real-world very well. For example, an event could be a notification that the temperature of a sensor has crossed a threshold. Again, these change notifications or events need to be distributed to all interested systems.
  • A system of microservices: An event could be a change or result emitted by one microservice and consumed by another. In these systems, each microservice needs to act independently and send change notifications asynchronously. Microservices should not be so tightly coupled as to know who specifically needs to receive notifications, but should instead explicitly describe which object has changed and how.

Event Topic

To achieve the promise of governed, enterprise-wide distribution of reusable events and event-driven architectures, the events need to be addressable and routable to the interested applications. This addressing is achieved via the event topic. The event topic is simply additional information (or metadata) in the messaging layer header. Topics are used for several purposes, but at their most basic they allow data processing systems to make intelligent routing decisions without deserializing, decoding, and interpreting the entire event. Intermediary nodes (event brokers of the system do not need to understand the entire event to take action; they just need to know how to act on the event's topic information. This is similar to the way IP routers direct internet traffic without inspecting the payload or even the upper level headers. However, unlike a URL or IP address, the event topic does not describe a destination, but instead describes the content of the message payload.

Event Subscription

While the event topic is metadata attached to the published event to allow for movement through an event mesh, the subscription is the mechanism for client applications to register interest in events. Upon matching event topics to consumer subscriptions, the system can weave dynamic forwarding paths across the mesh to deliver the correct events to each consumer.

Benefits of a Rich Topic Architecture

These key benefits result from a well-designed topic architecture:

  • Routing, filtering, and governance can occur at multiple levels as data flows across the event mesh. For example, a decision can be made on which events are allowed to cross geographical boundaries, but once a topic is in the geography, secondary decisions can be made on who can receive the events.
  • It's easier to gather information across events. For example, wildcards simplify the ability to receive all event types pertaining to an orderID, or all new orders regardless of originator or location.
  • There is an easily understandable relationship between the event and the business functional domain. For example, the topic hierarchy could contain levels from business unit, to application domain, to action; you might see a hierarchy like finance/payroll/pay-adjustment.
  • It removes the explicit need for orchestration. With proper eventing and flow of data, complex tasks can be choreographed and, with eventual consistency, completed in a highly scalable, robust way. For example, a new order for an online store triggers an inventory data change, a payment or billing change, and a shipping change. Any failure in inventory or billing causes a shipping cancellation. This set of events allows the billing, inventory, and shipping apps to scale independently and execute in an asynchronous manner.
  • Barriers to data movement (locality and source silos) are removed, while positive data governance is maintained. An enterprise-wide topic architecture allows events to flow across application domains.
  • Data versioning reduces the risk of regression, allows the development of canary application instances, and enables blue/green deployment scenarios.
  • Adding metadata properties to the unique topics that are published allows for fine-grained filtering, so consumers get only the events they need.
  • Adding SpanID and TraceID fields to the event hierarchy provides system-wide traceability and easier understanding of the data flow.
  • Multiple topic levels allow you to create access control rules that are level-specific and contain topic wildcards. This flexibility makes it simple to specify data entitlements at each level of the hierarchy.

Topics and Subscriptions in the Event Mesh

Topics and subscriptions serve several purposes in the event mesh.

Event Routing

To determine the path used to deliver events to interested consumers, the event broker inspects the event topic (which describes the data within the event) and matches it to the consumers' event subscriptions (which describe what data the consumer is interested in). The event topic provides a mechanism to make forwarding decisions based on information about the data without deserializing, decoding, and interpreting the entire data set. As described below, the event topic is similar to a URL for a REST message in that a URL allows intermediary nodes (firewalls and load balancers) to route messages to a specified destination. Event routing is different from REST-style point-to-point routing where the sender specifies the destination in that the event producer specifies only the routing topic, and the intermediary nodes (event brokers) route to all eligible destinations.

Event Filtering

Systems can have either managed subscriptions or consumer self-serve subscriptions. These subscriptions add fine-grained filtering to allow consumers to receive exactly the data they need from the mesh. Without this fine-grained filtering, intermediary processes in the system would be responsible for receiving, filtering, and re-publishing data. Because this logic can be implemented within the topic hierarchy and subscription filtering, adding intermediary processes for the specific purpose of applying business logic filtering adds unnecessary complexity and frailty to the overall system.

Event Access Control

Access control policies allow you to administratively manage which producers can publish on a topic and which consumers can subscribe to a topic, thereby controlling the movement of data.

Event Handling Policy

Based on topics and subscriptions, you can define policies that handle events within the event broker with respect to priority, availability, replication, and disaster recovery. The routing from producer to consumer can also be influenced by statically defined bridging policies.

Best Practices for Designing a Topic Hierarchy

Topic hierarchies are built in a very similar manner to the way the science of taxonomy classifies living entities into domains, kingdoms, classes, orders, and so on. These same principles can be applied to any large data set and provide a similar benefit—the ability to simply and logically drill down in a large data set to the exact element in which you have interest. In taxonomy, this is achieved through rank, where a species is described with a series of increasingly specific categories.

Without using messaging, you are already using hierarchical addressing today. If you go to a page on a website, the URL you enter is a hierarchical address with the form

application://sub-domain.domain/root-directory/sub-directorty/page

For example:

https://solace.com/use-cases/industries/retail/

The scale of the Internet would never work without the hierarchical addressing used by devolved and dynamic global DNS and web servers, which divide up the task of serving portions of the entire domain to end users. Similarly, if you have ever searched for a file in a file system, you have likely used wildcarding in a hierarchical addressed system. For example:

ls /home/solace/use-cases/i*

Event Topic Structure

Keeping with these concepts, we want to produce hierarchical topics that rank from least specific to most specific. To decide whether to add a field into a topic, consider whether it would it be used for routing, filtering, access control, or handling policies. If it's not likely to be used for any of these purposes, it doesn't need to be in the topic. Next, try to determine which fields are more specific than others by thinking about variability. The most variable fields are usually the most specific. This naturally leads to a topic hierarchy that has the most variable items at the end of the topic string. This in turn makes it much easier to use wildcarding as a method of grouping like topics.

Consider the following examples:

  • If we're creating a topic hierarchy for an event system that updates city bus positions, we could have a hierarchy like objectID/location. In this case, the objectID could be the bus route number and the location could be the bus stop number or latitude and longitude coordinates. There are many more bus stops or possible geographic coordinates than there are bus routes, so the location is the more variable field.
  • If we're building a topic hierarchy for an event system that updates purchase orders per city, we could have a hierarchy like location/objectID. In this case, the objectID could be either the purchase order or the SKU of the purchase, and the location could be picked from an enumeration of possible cities. In a successful business there are (hopefully) more sales than sales locations, so in this case the objectID is the more variable field.

For the rest of this document we have selected a common topic structure and hierarchy, but this might have to be modified to meet your use case. It is important, though, to remain consistent within an application domain once you've decided on your hierarchy.

Parts of the Event Topic

The event topic structure has two parts:

  • The event topic root contains enough information to describe the type of event that has occurred. Each event topic root is a static field that describes the type of event. The list of event topic roots forms a catalog of events that can be produced and consumed. This catalog could be brought into the PubSub+ Event Portal event catalog, listing each event type along with details about the event. Each event topic root describes the event in as much detail as necessary to map it to a single data schema.
  • The event topic properties are optional fields that further describe a particular event. This part of the topic has fields that are dynamically filled when the producer publishes the event. These fields are used to describe the specific or unique attributes of this event instance that would be used for routing and filtering.

Event Topic Root

The event topic root of an event should have the following form:

Domain/ObjectType/Verb/Version/

The fields in the event topic root are described in the following table:

Field Description

Domain

The organizational element responsible for the system. Be careful to ensure that these values can survive internal organizational changes.

Examples of domains could be:

  • CompanyName/BusinessUnit/ApplicationDomain (for multi-vendor systems)
  • BusinessUnit/ApplicationDomain (for integrations between business units)
  • ApplicationDomain (for simple applications or integrations within a single business unit)

ObjectType

Identifies the type of the object (similar to the way a REST resource has a type; then the ObjectID listed below would be similar to the Rest resource's href or ID).

Examples: customer, customerAccount, product, payment

Verb

Describes the action that has been taken on the object. This is similar to the REST verbs POST, GET, PUT, PATCH, and DELETE, but it describes an action that has taken place.

This field is typically a past tense verb.

Examples: created, deleted, exceeded, rejected

Version

The version of this topic hierarchy. This field can be used for routing purposes, and is suitable for distinguishing major( and likely non-backward compatible) changes to the topic hierarchy. It also enables blue/green or canary deployments. The regular production consumers might subscribe to version 1, while canary consumers subscribe to version 2. There could be a separate version number within the event schema itself that would follow semantic versioning to indicate the backward compatibility of changes to the event payload structure.

Event Topic Properties

The event topic properties should have the following form:

Locality/SourceID/ObjectID

The fields in event topic properties are described in the following table:

Field Description

Locality

The geographical or structural location at which the event occurred. This may require multiple levels, such as latitude/longitude or region/location, or could be just a single level.

SourceID

The ID of the person or application that produced the event. This field may be important for validating the authenticity of the event or for capturing the last message sent by a redundant sender. This ID could be as broad as a venue, for example the originating bank of a financial transaction, or as narrow as a specific IoT device, SAP system, or connection that the producer currently holds.

ObjectID

A unique identifier for the object instance that was acted on, for example a product SKU or order ID.

Complete Event Topic

Putting together an event topic root and event topic properties creates an event topic that describes the event with a series of fields from least specific to most specific.

Domain/ObjectType/Verb/Version/Locality/SourceID/ObjectID

Keep in mind that these are best practices that need to be tailored to your unique development environment, and that any level can be skipped or expanded to multiple levels. Once you have made a decision on the topic architecture, it must be applied consistently across the object-type domain. The levels of the topic must be used systematically and have unchanging meanings, otherwise routing and filtering will provide inconsistent results. That said, don't skimp on the root of the topic structure; being too specific too quickly often leads to topics that can seem out of context to read and can make the merging of systems very difficult.

Other Topic Levels

There could be several other event metadata fields that would likely not be used for routing and filtering, but might be important for other uses such as message replay or special policy rules. These values could be additional topic fields, message header properties, or additional fields within the message body, depending how they are used. If they are included in the topic, they should be inserted with the same rules on variability which would determine where in the topic hierarchy they would be placed.

Some examples of additional topic levels are listed in the following table:

Field Description

HandlingInstruction

This field could include information like encoding, for example json or protobuf, where the same event has different encoding for different clients. This representation for data could be included in the event topic root, when there are multiple schemas for the same event type. Alternatively, this field could represent a message priority or other information that event handling policies might key on, for example indicating disaster recovery routing. For a more in-depth discussion of a topic hierarchy used for data replication, see Configuring Replication.

TraceID

Used for tracing, as defined in the OpenTracing Semantic Specification, consisting of fields such as ParentID and SpanID.

ParentID: The ID of the immediate parent process in the microservice chain.

SpanID: Anything that uniquely identifies the event itself. In simpler systems it could by a GUID or even excluded entirely if the significance of an individual event does not warrant tracking. An example would be IoT sensors or web clicks where it's not an individual event that warrants action, but instead the rate of events or rate of change. Inclusion or exclusion of SpanID would be analogous to data that would be written to a relational database with a row primary key versus data that would be written to a time series database with no row key. If there is no SpanID there also likely would be no TraceID, and the ObjectID would be the last field in the topic.

These fields provide a lineage of events. With these fields in place, a query for all events related to a root event provides traceability of events and actions throughout a distributed system. All child events would contain the originating TraceID. If the current process is the one creating the event, the TraceID and SpanID could be the same value.

For example, the originating purchase order in an online store would be the originating event. Then subsequent billing events would contain the original purchase orderID as the TraceID.

You should consider adding this functionality into the topic hierarchy only in environments where end-to-end tracing is required and resources are allocated to performing this function; it is likely not worth doing for ad hoc tracing. For example, using a TraceID might makes sense if you are setting up to trace the latency or reliability of individual orders through a system.

Including the Environment in the Topic Hierarchy

Do not use an environment designator (for example, dev, qa, prod) in your topic hierarchy.

Using an environment designator (for example, dev, qa, prod) in your topic hierarchy causes several problems:

  • It forces your application code to change as you promote from your development environment into pre-production or production. At minimum, you would need an environment variable or some other way for publishers to put the right environment into the topic, meaning that your configuration would not be not static.
  • It requires you to change your configuration in many areas, such ACLs, subscriptions, message replay, replication, distributed tracing, and so on. This means that your infrastructure as code and your CI/CD pipelines would not follow best practices because they would have to be changed between environments.
  • It prevents you from using Event Portal to track the promotion of events across development environments. This is because Event Portal conforms to accepted infrastructure as code methodologies, where you promote a single artifact across your development environments.

Putting the environment designator into a topic level is an anti-pattern and should be avoided.

Topic Architecture Case Studies

The following are few examples that show how to apply topic architecture best practices for some common use cases.

Case 1: Event-Driven Microservices, Choreography, and Event Sourcing

The purpose of implementing microservices is to break big, monolithic applications (services) into smaller, more manageable components (micro services). Making those microservices event-driven makes them more efficient, performant, scalable, and robust.

The two key attributes of event-driven microservices are:

  • choreography, where microservices react to changes in their environment. This is in contrast to the orchestration approach, where the system relies on either tight linkages between microservices or a central orchestrator that, although it drives microservice interactions, is also the main bottleneck that prevents scaling.
  • event sourcing, which allows microservices to function and recover without needing atomic transactions with two phase commits, for example. Consistency in the system is achieved by replaying specific events rather than managing the state of tightly-coupled services.

To learn more, see Event-Driven Microservices and The Architect’s Guide to Event-Driven Microservices on Solace.com.

To see how to design a topic hierarchy for event-driven microservices, let's look at the "Sol-Beer" example in the Messaging Patterns for Event-Driven Microservices blog post.

As the article explains, the removal of the point-to-point chaining of services has created a scenario very similar to the Middle Office example below; not only are all the applications or services acting on the same event, but they also drive toward eventual consistency. The difference here is that there could be follow on events generated by some services that gate or otherwise affect other services. For example, it might be the situation that the delivery vehicle can't be dispatched until the billing clears. This example shows the interrelationships between different topic domains, where some events can be processed independently but others are blocked. In our example, although you can't dispatch a delivery vehicle until after the billing clears, you can notify the delivery vehicles that there is an order being processed in their area.

Let's see how a well-defined topic architecture enables both the flow of events through this system and the flexibility to expand data set. Originally the system delivered beer by car, but as business grows, the company could add other products, like Scotch whiskey, or new delivery modes (delivery by drone), so they need to plan for business growth.

In this system, customer-generated events are injected into the event mesh by the Sol-Beer WebApp. The customer places an order and then wants to know that their order is being processed, when their beer is on its way, and so on, but they don't want any details about logistics or inventory levels.

We start with the following topic event template :

Domain/ObjectType/Verb/Version/Locality/HandlingInstruction/SourceID/ObjectID

The topic hierarchy for this interaction could be:

orders/{productGroup}/{action}/{version}/{area}/{deliveryMode}/{customerID}/{productID}/{orderID}

The fields in this hierarchy are described in the following table:

Field Description

orders

Literal value specifying the topic domain; describes the total of all orders events.

{productGroup}

A value from an enumeration of possible product group types (beer, scotch, and so on).

{action}

A value from an enumeration of order actions, for example initialized, updated, canceled.

{version}

Version number for this release of orders.

{area}

General region for which an inventory manager is responsible.

{deliveryMode}

A value from an enumeration of car or drone for the mode of transportation.

{customerID}

The customer identifier.

{productID}

The specific item being ordered (dark beer, light beer, two-year old scotch, etc.)

{orderID}

The unique order identifier.

The event catalog entries, which are derived from the event topic root, would look like the following:

  • orders/beer/initialized/v1
  • orders/beer/updated/v1
  • orders/beer/canceled/v1
  • orders/scotch/initialized/v1
  • orders/scotch/updated/v1
  • orders/scotch/canceled/v1

The properties of the topics would provide more metadata for routing and filtering.

For example, the inventory management system may be sharded to handle specific products or priority. An inventory manager responsible for beer distribution in the Ottawa area would subscribe to:

orders/beer/*/v1/ottawa/>

Notice the use of wildcards, indicating that the subscriber does not care about specific fields. Here, the inventory manager does not really care about individual order actions or delivery modes, so uses wild cards in the subscription to just include them all.

All intermediary steps carry through the customerID and orderID, which allows tracing and replay based on a single order or customer. This functionality is key for event sourcing.

For example, the billing events would also contain customerID and orderID so that billing events could be traced back to order events:

            billing/{productGroup}/{action}/{version}/{area}/{deliveryMode}/{customerID}/{productID}/{orderID}
        

The result event catalog would be:

  • billing/beer/paid/v1
  • billing/beer/nofunds/v1
  • billing/scotch/paid/v1
  • billing/scotch/nofunds/v1
  • ...

The event topic properties would look like:

ottawa/drone/cust4321/product456/order12345

The inventory manager and the delivery vehicle that need to know when the billing has cleared would subscribe to:

billing/beer/paid/v1/ottawa/>

After the beer purchase has been authorized and the driver or drone assigned, the customer can track delivery with live updates.

The delivery topic hierarchy could be:

deliver/{vehicleType}/{action}/{version}/{latitude}/{longitude}/{vehicleID}/(orderID)

The fields in this hierarchy are described in the following table:

Field Description

deliver

Literal value specifying the topic describes the total of all delivery events.

{vehicleType}

A value from an enumeration of vehicle types, for example, car, drone, and so on.

{action}

A value from an enumeration of order actions, for example enroute, delivered.

{version}

Version number for this release of deliver.

{latitude}

The latitude of the delivery vehicle position.

{longitude}

The longitude of the delivery vehicle position.

{vehicleID}

The delivery vehicle identifier.

{orderID}

The unique order identifier.

The resulting event catalog entries would be:

  • deliver/car/enroute/v1
  • deliver/car/delivered/v1
  • deliver/drone/enroute/v1
  • deliver/drone/delivered/v1

The event topic properties would look like:

45.421532/-75.697189/vehicle4321/order12345

An end customer tracking the delivery of their specific order would subscribe to:

deliver/drone/enroute/v1/*/*/vehicle4321/order12345

A fleet manager tracking all their vehicles within an area might subscribe to:

deliver/*/enroute/v1/45*/-75*/vehicle4*/>

Flexible subscriptions allow events to be reused in ways that were not previously planned. For example, if new regulations came into place where commercial drones need to be tracked and reported to the city, a reporting agent could subscribe to the geographical area for which it is responsible:

deliver/*/enroute/v1/45*/-75*/>

Case 2: Bimodal Development From a Single Source of Truth

In this example there is a need to extract events from a system of record, or single source of truth, and distribute them to multiple downstream applications. The system of record could be a database or third-party application, but the pattern is the same. The key is that instead of each downstream application polling the system of record for changes, the system of record emits one event per change and all subscribing downstream applications have the ability to consume the event. This reduces the workload on the system of record. As shown in the diagram below, when a new purchase order is created in an SAP system, ASAPIO can push an event notification out to a PubSub+ event mesh, which streams that notification out to multiple interested applications (on-premises and in the cloud), like those associated with billing, manufacturing and support.

For more information about this SAP-specific system of record use case, see Solace with ASAPIO.

With ASAPIO there can be different encodings (JSON, XML, etc.) of the same event. Since there can be only event schema per event, we need to treat each encoding as a separate event. Because of this, we promote the HandlingInstructions field into the event root before the Version field. The result is the following event topic template:

Domain/ObjectType/Verb/HandlingInstruction/Version/Locality/SourceId/ObjectId

Let's see how a well-defined topic architecture helps data flow throughout this complex system.

In this use case, an event is emitted each time certain business objects (BO) change in the SAP system.

The events flowing from SAP to downstream applications would have one of three different object types or "aspects", each with progressively more information, which would be controlled through ACLs. These aspects are described in the following table:

Aspect Description

sap-bo-event

A notification only that a particular business object (BO) has been created or updated. The topic and payload contain enough information for an application or user to retrieve the business object according to their SAP access rights.

sap-bo-metadata

Contains the information from sap-bo-event plus the IDs of the fields that have been updated. This allows an application to further filter by fields they are interested in before retrieving the business object.

sap-bo-data

Contains the information from sap-bo-event plus the actual fields and the changed values. This allows an authorized application to instantly get the changes with their values.

For these SAP events the topic template would be as follows:

sap/{resourceType}/{aspect}/{action}/{representation}/{version}/{siteID}/{orgID}/{systemID}/{resourceID}
Field Description

sap

Literal value specifying that these are SAP-originating events.

{resourceType}

The business object type that the notification applies to. For example, BUS2032 is a sales order.

{aspect}

A value from an enumeration of object types sap-bo-event, sap-bo-metadata, or sap-bo-data.

{action}

A value from an enumeration of actions, for example create or update.

{representation}

The payload format: JSON, IDOC, XML, ARIBA-XML.

{version}

Version number for this release of the sap topic hierarchy.

{siteID}

The siteID, plant-ID, or location-ID where this BO is owned, where GLOBAL indicates a global entity, and EMPTY indicates that there is no siteID associated with the notification.

{orgID}

The organization unit that owns the business object, where GLOBAL indicates a global entity, and EMPTY indicates that there is no orgID associated with the notification.

{systemID}

The SAP instance or system ID.

{resourceID}

The business object ID.

The event catalog entries, which are derived from the event topic root, would look like:

  • sap/BUS2032/sap-bo-event/create/JSON/v1
  • sap/BUS2032/sap-bo-event/update/JSON/v1
  • sap/BUS2032/sap-bo-metadata/create/JSON/v1
  • sap/BUS2032/sap-bo-metadata/update/JSON/v1/
  • sap/BUS2032/sap-bo-data/create/JSON/v1/
  • sap/BUS2032/sap-bo-data/update/JSON/v1/

To see how this would be used, consider the event:

sap/BUS2032/sap-bo-data/update/JSON/v1/empty/1000/ERD100/0000096852

As we read through the fields in the topic hierarchy, we see the following information:

  • SAP event
  • Business Object: BUS2032 (sales order)
  • Object Type: sap-bo-data (contains actual data)
  • It's an update to a business object
  • Encoded in JSON
  • API version: 1
  • Site ID: none
  • Organization ID: 1000
  • SAP instance: ERD100
  • Business Object ID: 96852

A downstream application that wants to receive all sales order updates with their data from organization 1000 would subscribe to:

sap/BUS2032/sap-bo-data/update/JSON/v1/*/1000/>

Note that this application would need the appropriate access permissions (via administratively applied ACL) to allow it to subscribe to sap-bo-data events from organization 1000.

A downstream application that wants to receive all sales order updates (event only) from any SAP instance, any organization, at any site would subscribe to:

sap/BUS2032/sap-bo-event/*/JSON/v1/>

Again, this application would need to have the appropriate access permissions to allow it to receive notification events of new sales orders.

Case 3: Data Ingestion and Distribution

In this example there are data sources throughout the enterprise writing data into distributed data stores. These data sources could be published transaction logs from a database, online purchases, market data orders, and so on.

The diagram below illustrates the example of market data orders and post trade data, where the trade itself takes place in the "Front Office" and the post-trade activity takes place in the "Middle Office". Here we can see the client gateways produce events for the Front Office and the trading platform is the producer of events that flow to the Middle Office systems. We can also see the Middle Office is not a single-purpose consumer application, but instead is a series of applications that each have specific tasks and data requirements.

For details about this use case, see Investment Banking and the Middle Office Shock Absorber on Solace.com.

Without a properly designed event topic architecture that allows event re-use, you might have a very expensive trading platform publishing the same event multiple times in multiple formats to satisfy each Middle Office need.

We start with the standard topic event template of:

Domain/ObjectType/Verb/Version/Locality/HandlingInstruction/SourceID/ObjectID

Let's see how a well-defined topic architecture helps data flow throughout this complex system.

Front Office

The message exchange pattern in this phase of the interaction consists of a simple command message. As shown in the diagram above, the data flow for the Front Office works like this:

  1. A Client registers an intent to buy or sell a product with a Client Gateway.
  2. The Client Gateway turns this request into a Client Order and sends it to a Trading Platform.
  3. The Trading Platform determines the appropriate Exchange to use, and places the Market Order.
  4. The Exchange fulfills the Market Order and sends confirmation back to the Trading Platform.
  5. On receipt of a Fill event from the Exchange, the Trading Platform notifies the Client (via the Client Gateway) that its order has been filled.

For the Front Office, the topic template for client orders would be as follows:

orders/{productGroup}/{action}/{version}/{market}/{source}/{product}
Field Description

orders

Literal value specifying the topic describes the total of all orders events

{productGroup}

A value from an enumeration of possible product groups.

{action}

A value from an enumeration of order actions, for example new, update, cancel.

{version}

Version number for this release of orders

{market}

General region or group of markets for which the trading platform is responsible

{source}

Customer ID

{product}

Specific item being ordered.

The event catalog entries, which are derived from the event topic root, would look like:

  • orders/equity/new/v1
  • orders/equity/update/v1
  • orders/equity/cancel/v1
  • orders/bond/new/v1
  • orders/bond/update/v1
  • orders/bond/cancel/v1
  • ...

The properties of the topics provide more metadata for routing and filtering. For example, the Smart Order Routers (SORs) may be sharded to subscribe to subsets of the products to allow horizontal scaling.

The event topic properties might look like:

us/pri1/customer1/appl

To handle large volumes and possible volume spikes, the SORs split the traffic, first by productGroup and then at lower levels to handle shards or the product space.

The first SOR instance could subscribe as follows:

  • orders/equity/*/v1/NA/*/a*/>
  • ...
  • orders/equity/*/v1/NA/*/m*/>

And the second could subscribe as follows:

  • orders/equity/*/v1/NA/*/n*/>
  • ...
  • orders/equity/*/v1/NA/*/z*/>

Monitoring apps could track individual customers using the following subscription:

orders/equity/*/v1/NA/custID4321/>

Actions like cancellations could be monitored to flag unusual activity:

orders/equity/cancel/v1/NA/>

The SORs would then publish to a topic that includes the specific Exchange the trade should occur on, for example:

transaction/equity/new/v1/NYSE/traderID1234/Order4321

The Exchange Gateways would subscribe to the exchanges and pools they service, for example:

transaction/equity/new/v1/NYSE/>

Middle Office

After the Trading Platform has completed the trade, it sends an event message into the Middle Office system. The Middle Office likely includes several systems that must have a consistent view of the trades that happened, but where each needs specific data to perform its tasks.

transaction/{productGroup}/{action}/{version/{venue}/{source}/{orderID}
Field Description

transaction

Literal value specifying the topic describes the total of all orders events

{productGroup}

A value from an enumeration of possible product group types

{action}

A value from an enumeration of transaction actions such as new, updated, filled, canceled.

{version}

Version number for this release of transaction.

{venue}

A value from an enumeration of the liquidity pools or exchanges where transactions can occur.

{source}

Customer ID

{orderID}

The identifier for the unique product order

In this example, you could imagine that applications like Settlements and Risk would be interested only in successfully completed transactions, whereas Reporting might need access every transaction including canceled ones and Trade Data Capture might consume both every order event and every transaction event, either the entire set of data or again sharded by product type, region, venue, or product.

For example, with events like:

transaction/equity/new/v1/NYSE/traderID1234/Order4321

Settlements might subscribe to:

transaction/equity/filled/v1/>

or:

transaction/equity/filled/v1/NYSE/>

Whereas Risk might subscribe to the following to monitoring the positions of a particular trader:

transaction/equity/*/v1/*/traderID1234/>

Case 4: IoT Devices—Command, Control, and Analytics

Traditional IoT use cases involved exclusively data movement from the field sensors into back end state capture or analytics systems. The initial goals where simple systems that could horizontally scale to handle millions of devices. IoT uses cases have evolved as the numbers and complexity of the end devices have evolved. Security is now mandatory as well Command and Control of end devices is much more popular within these systems. This is because the events coming off IoT sensors are counted on for more than simple statistics and drive changes in upstream systems. Some critical sensor events might require human interactions, so it is important the events are accurate and valid. Think about air-bag deployment event on a mobile vehicle and the consequence if these events can be spoofed or ignored.

To learn more about the connected car use case, see the Internet Of Things article on Solace.com

Starting with the topic event template of:

Domain/ObjectType/Verb/Version/Locality/HandlingInstruction/SourceID/ObjectID

Let's see how a well-defined topic architecture helps authorized and route events from remote device to back-end systems, and how back-end systems can control remote devices.

Device to Back-End Services

Events emitted by the end IoT devices would consist mainly of sensor data.

Topic Format:

fromVehicle/vehicleComponent/action/version/{severity}/{username}/{sensorID}
Field Description

fromVehicle

Literal value specifying data direction.

{vehicleComponent}

A value from an enumeration of controllers or apps, for example TCU, MCU, and so on.

{action}

A value from an enumeration of sensor actions (reset, update, failure). Different vehicle components may have different actions.

{version}

Version number for this release of fromVehicle.

{severity}

A value from an enumeration of severity, for example error, warning, info.

{username}

Username or device ID used to validate authorization to produce event.

{sensorID}

The unique sensor that produced the event.

The event catalog entries, which are derived from the event topic root, would look like:

  • fromVehicle/tcu/reset/v1
  • fromVehicle/tcu/update/v1
  • fromVehicle/tcu/failure/v1
  • fromVehicle/mcu/reset/v1
  • fromVehicle/mcu/update/v1
  • ...

Then the properties of the topics would provide more metadata for routing and filtering. For example, the location of the vehicle might affect event routing and the severity of the event might affect the number of applications interested in the event.

Let's say a vehicle has an accident and its airbag deploys. The vehicle might emit an event with topic:

fromVehicle/tcu/update/v1/critical/vin4321/airbag

There could be an application listening to all critical severity events:

fromVehicle/*/update/v1/critical/>

In many cases, the IoT devices are not in a secure location and can be subject to physical or software hacking. For this reason, the device must be authenticated on every connection, and then the authenticated ID must be included and validated with each message to ensure the device only sends authorized events. This is the reason the MQTT username field is so important in this type of environment—it prevents one IoT device from pretending to be another IoT device.

For example, verifying the username would prevent a hacked vehicle with username vin4321 from sending an event as vin1234. The following update from username vin4321 would be allowed:

fromVehicle/tcu/update/v1/critical/vin4321/airbag

However, the following update from username vin4321 would not be allowed:

fromVehicle/tcu/update/v1/critical/vin1234/airbag

Back-End Services to Device

The traffic from back-end applications to vehicles could consist of multicast messages to several vehicles or unicast messages to a specific vehicle. Multicast messages could be based on location, on-board application, vehicle type, or criteria. For unicast messages, the username ensures that only the intended vehicle receives the update.

Topic Format:

toVehicle/vehicleComponent/action/version/{location}/{clientID}/{sensorID}
Field Description

toVehicle

Literal value specifying the data direction.

{vehicleComponent}

A value from an enumeration of the controllers or apps, for example TCU, MCU, and so on. An MCU could act as multiplexor to downstream sub-components.

{action}

A value from an enumeration of actions (possibly dependent on the vehicle component).

{version}

Version number for this release of orders

{location}

Multicast area, group, or null.

{username}

The username or device ID used to validate whether the sender is authorized to produce the event.

{endpointID}

The ID of the specific sub-component or app that should receive the control message.

The event catalog entries, which are derived from the event topic root, would look like:

  • toVehicle/tcu/update/v1
  • toVehicle/mcu/update/v1
  • toVehicle/weather/update/v1
  • toVehicle/traffic/update/v1
  • ...

Then the properties of the topics would provide more metadata for routing and filtering.

The vehicle would subscribe to all events directed specifically to it:

toVehicle/*/*/v1/*/vin4321/>

The vehicle could also subscribe to things like weather updates in its region:

toVehicle/weather/update/v1/ottawa/weatherPlan4321/>

Or it could subscribe to traffic updates for the current road and direction:

toVehicle/traffic/update/v1/417east/trafficPlan4321/>

As previously mentioned, the username is used for access policies and filtering. For a detailed discussion about access control in IoT applications, see Improving IoT Security with Access Control List Substitution Variables.