Topic Architecture Best Practices

Event-driven architecture, event-driven microservices, and event-driven integration are valuable application design patterns. The key advantage of these systems is that applications can collaborate by asynchronously reacting to events instead of relying on synchronous polling or orchestration for signaling. The publish-subscribe message exchange pattern is a common way to deliver events. A well-architected topic hierarchy:

  • effectively routes events across an event-driven system and ensures consumer applications have the flexibility to selectively consume only the events they want
  • allows applications to efficiently attract and consume events and get the most out of Solace smart topics

A rich, well-defined topic architecture is key to maximizing the value of event-driven architecture. It offers many benefits for routing, filtering, and governance. In addition, the topic architecture can be designed and captured in PubSub+ Event Portal, allowing for event discovery and runtime management.

For more information and background about topics, see:

Topic Architecture Definitions

You should be familiar with the following event-driven architecture terminology.

Event

An event can be broadly described as something that happened within your organization, particularly a change in state of some kind of object. Events can have a variety of forms, but all have the common component of an action that has occurred on an object. For example:

  • In a system of microservices, an event is a change or result emitted by one microservice and consumed by another whenever a state update occurs. For example, a customer places an order or opens an account. In these systems, each microservice acts independently and sends change notifications asynchronously. Microservices are loosely coupled, so the publishers do not need to know who needs to receive notifications, but instead use smart topics to explicitly describe which object has changed and how.

  • In an IoT system, an event represents a state change of a real-world object. For example, an event could be a notification that the temperature of a sensor has crossed a threshold. Again, these change notifications or events need to be distributed to all interested systems.

  • In a data system, an event represents a create, update, or delete operation. As a result of a create operation, a backend may update the database and emit a completion event. The difference in event-driven architecture is that, unlike a database, the event mesh is an open and distributed system where the events can be distributed to all the interested applications. This removes the need for applications to poll for changes in the database.

Event Topic

To achieve enterprise-wide distribution of reusable events in event-driven architectures, the events need to be routable to interested applications. This routing is achieved via a smart event topic. A topic is additional information (or metadata) in the messaging layer header, which is set by the application that produced the event. Smart topics allow event brokers to make intelligent routing decisions without deserializing, decoding, and interpreting the entire event. Event brokers don't need to understand the entire event to take action; they just need to know how to act on the event's topic information. This is similar to the way IP routers direct internet traffic without inspecting the payload or even the upper-level headers. However, unlike a URL or IP address, the event topic does not describe a destination, but instead describes the content of the message payload. To get value out of the event topics, they must follow a well-defined topic taxonomy which clearly defines the various hierarchical topic levels set by the producing application.

Event Subscription

While the event topic is metadata attached to the published event to allow for movement through an event mesh, the subscription is the mechanism for client applications to register interest in events. Upon matching event topics to consumer subscriptions, the system can weave dynamic forwarding paths across the mesh to deliver the correct events to each consumer. Unlike event topics, a single event topic subscription can be configured to attract numerous event topics via wildcards, which allow for fine-grained filtering of events flowing through the event mesh.

Topic Taxonomy Example

At its core, an event is an action that has occurred or state that has changed on an object. The topic for an event should describe the event. A basic topic hierarchy, or more formally the topic taxonomy, must describe the object (noun) and the action taken or state changed (verb). Some additional properties may also be included in the topic to enrich the hierarchy. In simplistic terms, a good topic hierarchy takes the form Noun/Verb/Properties.

Consider the topic architecture for Example Airline (EA), who are designing a new application to track flights. The domain would be ops/flights/ for flight operations. The ObjectType in this case is a flight, thus flight would be the noun. Various different actions may occur on these flights, for instance, a flight might start boarding or be delayed. To differentiate between different flights, flightNumber can be included in the topic. Subscribers may also want to receive information for flights at specific airports, so origin and destination would be good candidates to include in the topic. Putting it all together, we get flight/[status]/[flightNumber]/[origin]/[destination] as a basic topic architecture.

Within the applications at Example Airline (EA), when a state change occurs, the following events are published:

  • Flight EA 1234 is boarding: flight/boarding/ea1234/jfk/ord
  • Flight EA 9999 is delayed: flight/delayed/ea9999/yow/sin
  • Flight EA 1010 took off: flight/wheelsUp/ea1010/ewr/ord

This topic architecture allows subscribers to receive events for:

  • All flights leaving JFK flight/wheelsUp/*/jfk/>
  • All delayed flights for the airline flight/delayed/>
  • All events for flight EA 1010 flight/*/ea1010/>

As time goes on, you may require more properties, event payloads may evolve, and new applications may come online. To ensure that the system is future-proof, each distinct event should be versioned. In addition, each event topic should be prefixed with its data owner or domain. The domain indicates which department owns the defined event. In this case operations (ops), as well as the application domain to which the event belongs, which in this case is flights. This practice allows for future growth of the enterprise, either by adding a baggage application within the ops department, or onboarding an entirely new booking department’s applications.

As such, it gives us a final topic taxonomy of ops/flights/flight/[status]/[version]/[flightNumber]/[origin]/[destination], where the v1 publisher may publish ops/flights/flight/boarding/v1/ea1234/jfk/ord for the boarding example above. The following events may then be added to an event catalog for discovery:

  • ops/flights/flight/boarding/v1/{flightNumber}/{origin}/{destination}
  • ops/flights/flight/delayed/v1/{flightNumber}/{origin}/{destination}
  • ops/flights/flight/wheelsUp/v1/{flightNumber}/{origin}/{destination}

Benefits of a Rich Topic Architecture

These key benefits result from a well-designed topic architecture:

  • Routing, filtering, and governance can occur at multiple levels, as data flows across the event mesh. For example, a decision can be made on which events are allowed to cross geographical boundaries, but once a topic is in the geography, secondary decisions can be made on which consumers can receive the events.

  • Gathering information across events is easier. For example, wildcards enable receiving all event types pertaining to an orderID, or all new orders regardless of originator or location.

  • When events are documented well, for instance, with PubSub+ Event Portal, a well-defined topic hierarchy makes onboarding new team members and new applications much easier.

  • Relationships between events and business domains are easy to understand. For example, the topic hierarchy could contain levels from business unit, to application domain, to action; you might see a hierarchy like finance/payroll/pay-adjustment.

  • Explicit need for orchestration is removed. With the proper use of event-driven architecture, complex tasks can be choreographed and consistently completed in a highly scalable and more robust way. For example, a new order for an online store may trigger an inventory data change, a payment or billing change, and a shipping change. Any failure in inventory or billing causes a shipping cancellation. This set of events allows the billing, inventory, and shipping applications to scale independently and execute in an asynchronous manner.

  • Barriers to data movement are removed by using an event mesh and a well-defined topic hierarchy because data can selectively flow between application domains, while data can still be governed and access-controlled. An enterprise-wide topic architecture allows events to flow across application domains. For example, a marketing application may want to receive operations events, and with a well-defined topic architecture and a few tweaks to access control, this can happen without operations having to make any changes.

  • Event versioning offers several benefits: it reduces the risk of regression, enables the development of canary application instances, facilitates blue/green deployment scenarios, and future-proofs the system against business requirement changes. New event versions can be added as requirements change, without risking existing applications even if non-backward compatible changes are necessary.

  • Adding properties to the topic hierarchy allows for fine-grained filtering. This filtering ensures that consumers only receive the events that they need. It also ensures the system is more efficient by reducing the workload on applications and keeping egress data costs down.

  • Multiple topic levels allow you to create access control rules that are level-specific and can contain topic wildcards. This flexibility makes it simple to specify data entitlements at each level of the hierarchy. For example, including a sensor ID in the topic hierarchy allows access control lists to use substitution expressions to ensure that one sensor cannot publish data pretending to be from another sensor.

Best Practices for Designing a Topic Hierarchy

A well-designed topic hierarchy should follow the topic structure that we outline. We recommend that you follow this structure for all event topics across your organization. To see how these best practices can be put into use, see Topic Architecture Case Studies.

Event Topic Structure

The first step to defining a topic hierarchy is to identify the business objects participating in the event-driven microservice or event-driven integration. Next, the key actions or state changes that can occur to these objects should be identified. The combined object and action is generally referred to as the topic root, along with other information such as the business domain, and the version of the event. Then, for each of these topic roots, the optional properties specific to the object or action that occurred should be identified. This additional information is generally referred to as the properties of the event. You can decide whether to add a field to a topic, consider whether it would be useful for routing, apply filtering, and enforce access control.

Consider the following examples:

  • If we're creating a topic hierarchy for an event system that updates the city bus positions, we could have a topic hierarchy of mobile/bus/locUpdate/v1/<routeNumber>/<busNumber>/<location> . In this example, the routeNumber, busNumber, and location are all properties of the locUpdate action. There are many more bus numbers than there are route numbers so busNumber follows after routeNumber. Similarly, there are more locations for a bus than the number of buses, so location goes after busNumber.
  • If we're building a topic hierarchy for an event system that updates purchase orders per city, we could have a hierarchy of store/order/created/v1/<locality>/<objectID>. In this case, the objectID could be either the purchase order or the SKU of the purchase, and the location could be picked from an enumeration of possible cities. In a successful business, usually there are more sales than sales locations, so in this case, the objectID is the property with more variability.

Parts of the Event Topic

A well-defined event topic has two main parts:

  • The event topic root contains enough information to describe the type of event that has occurred. Each event topic root is a static field that describes the type of event. The list of event topic roots forms a catalog of events that can be produced and consumed. This catalog can be captured in PubSub+ Event Portal by adding all of your events to Designer. Each event topic root describes the event in as much detail as necessary to map it to a single data schema.
  • The event topic properties are optional fields that further describe a particular event. This part of the topic has fields that are dynamically filled when the producer publishes the event. These fields are used to describe the specific, or unique attributes of this event instance that could be used for routing and filtering. Examples of this are the object ID, and location from the preceding city bus example, or the flight number, origin, and destination from the preceding airline example. It is not uncommon for these fields to also be encoded in the event payload, but only the fields that are helpful for routing, filtering, or governance should be included in the topic.

While the topic hierarchy should be rich, it's important that it also be concise. A topic is limited to a maximum of 250 characters, and 128 topic levels. The topic root containing the domain, noun, and verb and one or more important properties should use literal values rather than variables and should be concise. We recommend that you include properties in the topic root literals only if they are valuable for routing, filtering, or governance.

Event Topic Root

The event topic root of an event should have the following form:

Domain/ObjectType/Verb/Version/

The fields in the event topic root are described in the following table:

Field Description
Domain

Identifies the organizational element responsible for the system. This should describe the system’s responsibility (e.g., operations or booking), as well as clearly identify the event’s owner (e.g., op or hr). Using multiple domain levels ensures that event names do not collide as the enterprise grows, and ensures that a clear data owner can be identified for each event.

Encoding domain information into the topic hierarchy enables multiple domains to seamlessly share the same mesh, which in turn allows for collaboration between domains. It also enables the governance needed for multiple business units to share the same event mesh and provides clear ownership of the event, which is essential for effective reuse of events and ensures accessibility, evolution, and data quality of the events.

The Domain should clearly identify the organizational owner of the event, and should take the form dataSystem/applicationDomain(e.g., finance/payroll or marketing/rewards). The dataSystem should be as wide as possible to enable event reuse, and the applicationDomain should differentiate different potential applications within the same department. The organization name may be included in the form organizationName/dataSystem/applicationDomain to enable multi-vendor systems, and future-proof the Domain against acquisitions or mergers.

Examples: operations/flights, acme/rideshare/billing
ObjectType

Identifies the type of the object (referred to as the noun) being acted on. This object may be very general, for instance, an order, or it can be more specific if significantly different handling has to happen for different products (e.g., securities trades versus mutual funds).

Examples: customer, order, inventory, payment
Verb

Describes the action that has been taken or the state that has changed on the object and can be typical CRUD operations or another verb describing the action taken. The verb for the field is typically in past tense.

Examples: created, deleted, exceeded, pickedUp, paid
Version

Identifies the version of the event. This field can be used for routing purposes, and is necessary for distinguishing major changes to the topic hierarchy or major changes to the schema of the payload. Including the version in the topic hierarchy enables blue/green or canary deployments, for instance, the regular production consumers might subscribe to version 1, while canary consumers subscribe to version 2. This version should simply take the form v1, v2, etc.

Examples: v1, v2

Event Topic Properties

Beyond the event topic root, properties can be added to the event topic hierarchy to make the topic more granular. Properties are specific to each use case, and not applicable to all use cases. A property should be included only if it is useful for subscriber filtering, event routing, or access control. Some common properities might be location information, product, customer IDs, or specifiers for the type of data. You can include properties are are also present in the event payload, though we recommend that you include only the properties that are important for routing, filtering, or governance should be included in the topic.

Event topic properties should be ordered in terms of cardinality: from least specific to most specific. Consider potential event topic properties on a sale order event that encode both the city in which the sale occurred and the order ID of the placed order. There are likely more orders placed than cities in which the enterprise operates, therefore it would make sense that the city comes before the order ID in the topic hierarchy.

For each property in your topic hierarchy, it is also important to consider the structure of the data. Particularly, the value space of the property should be clearly documented, whether that be a freeform string, an enumeration, an ID format, an integer or a floating point number. For instance, if including an airport field in your event topic properties, the format for the airport code should be documented as well as information on where to find the most up-to-date list of airports used by your application.

Field Description
ObjectID

A unique identifier for the object instance that was acted on. There may be more than one ObjectID, for instance, if an order is placed by a customer, the orderId and customerId may both be included. Only the IDs that are helpful for routing, filtering, or governance should be included.

Examples: orderID, sensorID, customerID, routeNumber, productSKU

Some uses of ObjectID might include subscribing to receive all notifications for a particular route number, or using access control to only allow IoT sensors to publish updates for their own sensorID.
Locality

The geographical or structural location at which the event occurred. The local may be a single level like a country, region, or transit stop number, or may require multiple levels such as latitude/longitude, region/location, origin/destination, etc.

Depending on how specific the locality is, it may be included near the beginning of the properties section (e.g., country codes such as US or UK), or it may appear near the end of the properties section (e.g., coordinates such as latitude and longitude).

Some uses of locality might be for geofencing (received data via latitude and longitude), or ensuring data sovereignty laws are respected based on the region information.
Category

Often, data falls into certain categories. These categories might not be different enough to be differentiated in the ObjectType field described in the Topic Root, but may still be valuable information in the topic. For instance, a ride share app may have various vehicle types, or there may be specific follow-on actions for certain products and not others.

Some uses of Category may be only receiving updates for a certain subset of data rather than the whole set, such as providing reports for a certain aspect of a product.
HandlingInstruction(advanced)

In advanced use cases, additional handling information may need to be included in the event topic, such as information about the payload or importance of the event so that routing and filtering decisions may be made.

For example, if your system emits events onto the event mesh through a connector that converts data sources into event streams that have different encoding types, such as json or protobuf, it may be important to include this information in the event topic to differentiate between the two. A handling Instruction may be useful if you have separate instances of the application that process the events based on the encoding.

This field may be promoted to the event topic root (e.g., before the version field) if different handlings may need to be versioned separately. For instance, if the json encoding may change independent of the protobuf encoding, the json encoding can be version bumped without impacting protobuf consumers.

If the only users of HandlingInstructions are instructing consumers on how to handle the event, consider using message properties instead. HandlingInstructions should only be included if the field is helpful for routing, filtering, or governance, or in cases where using message properties is not possible. For instance, when using off the shelf connectors that do not support message properties.

 

Complete Event Topic

Putting together an event topic root and event topic properties creates an event topic that describes the event with a series of fields from least specific to most specific. This leads to a topic template of:

Domain/ObjectType/Verb/Version/Properties...

Keep in mind that these are best practices that need to be tailored to each individual use case, and that any level can be skipped or expanded to multiple levels if there is a benefit to routing, filtering, or access control. After a decision has been made on which fields are included or omitted from the topic architecture, it must be applied consistently across the object-type domain. The levels of the topic must be used systematically and have unchanging meanings, otherwise routing and filtering will provide inconsistent results. In addition, do not skimp on the root of the topic structure, particularly the domain; being too specific too early in the hierarchy often leads to topics that can seem out of context to read and can make the merging of systems very difficult.

Documenting an Event

Each event should be well-documented, and each individual topic level should be clearly defined in terms of the format of the data. Documenting events in this way is often referred to as cataloging. Cataloging events has the benefit of clearly stating which properties are available on an event, and how each individual property functions. It is also important for event discovery which allows existing events to be reused by new applications. All of this can be done in PubSub+ Event Portal

For each event topic level, the value space should be clearly defined (e.g., integer, floating-point number, enumeration, or string format). For example, a property may contain airport codes, and the list of possible codes should be enumerated or linked to a source of truth. Furthermore, the events within an organization should follow a consistent pattern, such as camelCase for all naming within the event topic architecture. Using camelCase is recommended because snake_case requires additional characters for the same semantics. These decisions should be documented so that all events in the system follow a consistent format.

Subscription Exceptions

PubSub+ event brokers support negative subscriptions, prefixed with an exclamation mark !, which exclude the specified topic from a larger subscription set. Consider the case where a specific flight number’s data needs debugging. It may be helpful to have one instance of an application handle that particular flight number, EX1234, and have another instance of the application handle all other flight numbers. The application instance handling the specific flight can subscribe to flight/ex1234, and the instance handling all other flights can subscribe to flight/* and exclude flight EX1234 with the subscription !flight/ex1234.

If you are designing a solution where special handling is always required, it's a best practice to have a separate topic level to encode the property. This ensures that the subscribers don't need to closely coordinate in order to prevent losing data. We also recommend enabling reject-msg-to-sender-on-no-subscription-match on client profiles that use negative subscriptions to ensure that excluded messages are not lost.

Subscription exceptions are supported only for Guaranteed messaging. For more information, see System-Level Subscription Exception Configuration.

Null Values

There may be occasions where a topic level may be null. If this occurs within your taxonomy, first consider if the field is still useful for routing, access control, or governance, and if it's not, simply remove the level from the taxonomy. If it is still valuable, consider if a reasonable default can be used. If not, a null value should be defined, both so your application can subscribe to any topics containing this value, and so you can apply access control and governance configurations to the topic.

Choosing a null value may be as simple as choosing a reasonable default, often referred to as a sentinel value. Some examples of sentinel values are:

  • defaulting to 0 for a numerical field
  • defining a default value for enumerations
  • defaulting to customer ID to 0000 when no customer is logged in

Free-form strings are the most complex and need to be considered on a case-by-case basis. _null_ is a good starting point if _ cannot occur naturally within the value space of the field. It is important to be aware of user input in this case, and ensure that the input cannot overlap with the null value (e.g., _ must be an invalid character in the user input). Special characters in SMF, such as #, _, *, and >, should be avoided in the chosen null value.

Regardless of the value chosen, whether sentinel or a special null, it must be documented, along with the event as a possible value for the field.

Topic Architecture Anti-Patterns

There are a number of patterns that should be avoided when designing a topic architecture. A few common anti-patterns are outlined.

Don't Use Message Properties for Filtering

Many legacy messaging systems rely on features, such as JMS selectors , to filter data going to consumers based on user-defined message properties. PubSub+ has legacy support for selectors, however filtering based on selectors is an anti-pattern. Selectors cause a number of architectural problems, such as mixing the responsibilities of filtering with event content resulting in complicating the event schema. Furthermore, access control, routing, and governance decisions cannot be made based on message properties. In general, selectors should be avoided, and the routing, filtering or governance information should be encoded into the topic hierarchy. Message properties should be used only to communicate metadata about an event to consumers of that event.

Don't Include Tracing Information in the Topic Hierarchy for Tracing Use Cases

Tracing information from an event, such as TraceID, SpanID, or any other tracing data, should not be included in the topic hierarchy. Instead, Solace native support for Distributed Tracing should be used along with a tracing backend to enable tracing use cases. Including the TraceID in the topic hierarchy uses up valuable space in the topic string, and provides less visibility than Solace Distributed Tracing. The TraceID is also not useful for routing, filtering, or governance because it is an arbitrary value that changes for each message.

Don't Include Deployment Environment Names in the Topic Hierarchy

Environment designators, such as dev, qa, or prod should not be used in your topic hierarchy. If you include environment information in your topic hierarchy, it causes several problems:

  • Forces your application code to change as you promote it from your development environment into pre-production or production. At minimum, you would need an environment variable or some other way for publishers to put the right environment into the topic, meaning that your configuration would not be static.
  • Requires that you to change your configuration in many areas such as access-control lists (ACLs), topic subscriptions, message replay configuration, replication configuration, distributed tracing, and so on. Consequently, your infrastructure as code and CI/CD pipelines may deviate from best practices and require modifications based on the target environment.
  • Prevents you from using PubSub+ Event Portal to track the promotion of events across development environments. It's important to recognize that Event Portal conforms to industry-accepted infrastructure as code methodologies, such as where you promote a single artifact across development environments.

Don't Include Spaces or Other Special Characters in Topics

Avoid using non-alphanumeric characters in the topic taxonomy, such as spaces and special characters. Including spaces makes the topic more difficult to read, and also makes the topics less reusable, as well uses up characters unnecessarily. Furthermore, *, > and ! should be avoided on any published topics as they are treated as literals when you publish the topic, but are treated as special characters in topic subscriptions. For this reason, it is very difficult to subscribe to events when topics contain these characters.

For similar reasons, avoid snake_case when possible because using separator characters is less efficient than other naming conventions. For this reason, you should use PascalCase or camelCase instead.