Event Stream Concept Maps

The Solace PubSub+ platform, which includes the appliance, software, and cloud products, provides high-performance global event routing for applications connected to a Solace event mesh.

It's important to know how events and messages move through PubSub+ event brokers - which is what we'll discuss in this section - and have some familiarity with power-user capabilities and specialized features that can be configured inside a broker. This will help you avoid creating code to mimic existing capabilities to solve design problems that have already been solved by the PubSub+ event broker's standard features.

Hierarchical Topics

One of the most important subjects you need to know in order to understand the movement of events through PubSub+, and defining the event stream you want available in your organization, is hierarchical topics, and wildcard masking against those topics. Once you've learned the basics of topics, you're well on your way to mastering PubSub+ technology. We recommend taking a look at Understanding Topics .

Of particular interest, PubSub+ offers a number of reserved or special topics that make configuration of a range of capabilities from things like queue routing, message promotion and demotion, and disaster recovery simple. You can find out more at Reserved Topics, and information about general topic syntax can be found atSMF Topics.

Persistent vs Durable

In the discussions that follow you need to understand the difference between the terms persistent and durable, so in this section we'll take a look at them before proceeding. One thing to keep in mind as you read through the definitions below is that the concept of persistence applies to event messages, and durability applies to endpoints.

Persistent

Persistence refers to the Quality-of-Service (QoS) of event messages, and they can be classified as Persistent (Guaranteed) or Non-Persistent

Event messages are considered to be persistent if they are placed onto non-volatile storage media after they arrive on the broker. The broker‘s data plane stores a message if the message arrives with the Persistent Delivery Mode set to Persistent, or if a queue or durable topic endpoint has subscribed to a topic where messages are persisted regardless of the Persistent Delivery Mode flag setting in the message.

If an event message is sent with Persistent Delivery Mode set to Direct, which is synonymous with Non-Persistent, the message is only placed onto the non-volatile storage media if the message is subscribed by a Queue or a Durable Topic Endpoint. Messages flagged as Persistent will result in an acknowledgment message sent back to the producer once the message is stored.

Durable

Endpoint Durability is a characteristic of a PubSub+ queue or topic endpoint. They are state engines that control reference to persistent messages in the Persistent Event Stream, which we'll discuss in the Persistent Event Stream Maps section.

Durable endpoints operate whether or not consumers are active. For example, queue messages will still be available after a broker fails, or there is a High Availability (HA) or Disaster Recovery (DR) fail-over. Therefore, messages are considered to be persistent in a general sense if they are associated with a Durable Queue because they stick around independent of the life span of particular client session or event broker restart.

Temporary queues aren't considered to be durable. Messages to temporary queues are persisted to storage, but they're deleted if the active temporary queue consumer unbinds or disconnects. Therefore, with non-durable endpoints, the endpoint is not durable beyond the life of the consumer.

You can find out more about durable and non-durable queues at Endpoints / Queues.

Non-Persistent & Persistent Event Streams

PubSub+ separates incoming events into two separate streams depending on the QoS required (as defined by the producer) for the events as they are moved from producers to consumers. These are called Non-Persistent Event Stream Maps and Persistent Event Stream Maps, and they are processed through different event stream paths. In the following sections we'll show you, step-by-step, how that processing takes place.

Non-Persistent Event Stream Maps

Non-durable consumers directly consume messages from the Non-Persistent Event Stream by simply attracting messages based on a consumer subscription request. As new event messages arrive at the broker, they are placed in published order onto the consumer’s egress queue based on the client’s subscription request.

Non-persistent events are ephemeral

Non-persistent events are ephemeral. These events provide a QoS for consumers where loss of messages is acceptable, and use a design pattern where consumers only receive messages starting from the time of connection. This allows for extremely high throughput and ultra-low latency as a trade-off against longevity. This QoS is also commonly used for Message Exchange Patterns (MEPs) such as Request-Reply or Publish-Subscribe for applications like market data.

Message arrival

When event messages arrive at the broker with Persistent Delivery Mode set to Direct, the message is placed at the tail of the Non-Persistent Event Stream.

Consuming messages

Non-durable consumers directly consume messages from the Non-Persistent Event Stream simply by attracting topic messages based on a subscription requested by the consumer. As new event messages arrive at the broker, they are placed in the Consumer’s Egress queue based on the client’s subscription request.

Message priority

Each Direct message has an assigned priority. The consumer’s egress queue ensures higher priority Direct messages are processed before lower priority ones. Messages of the same priority are processed in the order they arrive at the broker.

Non-persistent event processing

In the next map you can follow a non-persistent event as it's processed by PubSub+ :

  • Step through the processing of a non-persistent event
  • Hover over the or icons next to the circled numbers in the map below to see information about event processing at this location. Clicking the icon will guide you to further information about some aspect of the step.

  • Find out more about PubSub+ components
  • Clicking on icons associated with components will lead you to further information about those components.

     

1. Message producers send events with the header's Persistent Delivery Mode set to Direct. The messages are written to the tail of the Non-Persistent Event Stream based on the order they arrive at the broker. If a message is sent with the Persistent Delivery Mode set to Persistent, it will also be put into the Non-Persistent Event Stream for those consumers who are subscribed to the topic, but don't require message persistence. To understand how messages flow from producers to consumers you need to understand the role played by topics. <br> Click here for a high-level, API neutral, discussion of the fundamentals of publishing and subscribing to topics. 2. The arrival of messages in the Non-Persistent Event Stream causes the Data Plane Task to determine which Non-Durable Consumers are interested in the message. 3. A reference to the message is moved to the appropriate Consumer Egress Queues for all consumers whose topic subscriptions match the message's topic. Based on the priority of the messages, the messages are then queued in TCP for delivery to the appropriate consumer. 4. Once all consumers that have registered interest in the same topic message, and their Egress Queue schedulers have moved the message to the TCP output queue, the message is deleted from the Non-Persistent Event Stream. Each Direct Messages has a priority assigned. The Consumer’s Egress Queue ensures higher priority direct messages are processed before lower priority messages. Messages of the same priority are processed in the order they arrived at the broker. Click here for information on how to receive and manage Direct messages using messaging APIs. Click here for a discussion on publishing Direct messages. Data Plane Tasks for Incoming TCP Events

 

Data Plane Tasks for Incoming TCP Events

All messages enter a Solace PubSub+ broker via custom ingress TCP buffers. The arrival of new TCP event messages cause the Data Plane Tasks to move the TCP data to the Non-Persistent Event Stream for further processing. However, it should be noted that messages are only moved to the event stream if they pass ingress ACL processing.

Messages in the Non-Persistent Event Stream are ephemeral, and message discard is acceptable. All messages, including messages that are flagged with a Persistent Delivery Mode set to Persistent or Direct are first placed into the Non-Persistent Event Stream when they arrive over TCP/IP. The Data Plane Tasks immediately move messages flagged as Direct to the consumers’ egress queue where the consumer has a matching topic subscription.

Messages flagged as Persistent are immediately placed into the Persistent Event Stream. If the persisted mode messages are discarded before they are processed into the Persistent Event Stream, the producer API ensures redelivery to the broker, and the Persisted Event Stream will automatically discard redelivered duplicates.

Operations the Data Plane Tasks Perform on Messages Arriving over TCP

Once the messages pass the ingress ACL check, they are placed in the Non-Persistent Event Stream, and are further processed by the Data Plane Tasks.

  • Promotion
  • Some messages will have arrived with the Persistent Delivery Mode set to Direct, but a queue may have a topic subscription for those messages. Topic messages subscribed to from queues are promoted to Persistent, and moved to the Persistent Event Stream, but no ACK is sent to the producer.

    For more information about promotion, take a look at Message promotion.

  • Demotion
  • Some messages will arrive in the Non-Persistent Event Queue that have the Persistent Delivery Mode set to Persistent. However, there may be topic subscribers who want the same message while it's being moved to the Persistent Event Stream. This message is demoted to Direct for delivery directly to the consumer egress queue for ultra-low latency and high throughput processing for consumers who don't require guaranteed delivery.

    Further information about demotion can also be found at Message demotion.

  • Prioritization
  • Based on an indication from the producer, the message is placed into one of three consumer egress queue priority streams.

    For more information refer to Client Egress Queue Structure Overview.

  • Shared Subscriptions
  • Messages can be delivered in a round-robin fashion to a group of topic consumers to provide load balancing similar to non-exclusive queue processing.

    More information can be found at Shared Subscriptions.

  • Persisted Message moved to Persistent Event Stream
  • All messages from the TCP network are placed directly into the Non-Persistent Event Stream. If the message's Persistent Delivery Mode is set to Persistent, the message is moved and persisted against the Persistent Event Stream.

  • Logging
  • Event messages are logged in real-time based on multiple conditions ranging from threshold settings to messages arriving without any subscribers. This is a log of message activity, not tracing of the full message.

    Information about configuring logging for various APIs can be found in Configuring Logging.

  • Subscription Binding
  • When consumers indicate they want specific topic messages for Direct mode delivery (including demoted messages), the attracted topic messages are now referenced in the consumer’s egress queue for processing to the TCP outbound buffers.

  • Consumer Egress Queue
  • A reference to the messages in the Non-Persistent Event Queue that are scheduled for TCP delivery to specific consumers. The reference will only be added to the egress queue if the consumer’s security ACL profile allows the delivery.

    For more information refer to Client Egress Queue Structure Overview.

The Data Plane Tasks Map

By selecting the icon beside the various components in the map below, you can learn more about Data Plane Tasks.

 

Message promotion is the situation where a producer sends Direct messages, and the consumer receives these message from a Guaranteed messaging endpoint. Message demotion is the situation where the producer sends Persistent messages, and there are consumers that want to receive these messages, but can tolerate lost messages. When you enable an endpoint to respect message priority, the priority field in messages from producers are respected for all guaranteed and promoted direct messages. Shared subscriptions can be used to load balance large volumes of client data across multiple instances of back end data center applications. This section provides information for using messaging API logging. Typically, messages are published to a Queue when it's set as the destination of the message. However, you can also add a topic subscription to a Queue so that it receives any messages that are published to a matching topic destination. All messages from the TCP network are placed into the Non-Persistent Event Stream directly. Based on the Delivery Mode in the message set to Persistent, the message is moved and persisted against the Persistent Event Stream. You can use ACLs to control the topics to which clients are allowed to publish. Click here for information on egress topic ACLs Click here for information that describes the egress per-client priority queues on brokers, and the commands that you can use to configure them.

 

Non-Durable Temporary Queues

There are several MEPs where there's no requirement for a persistent queue to exist beyond the life of a durable consumer. These queues are called temporary queues, and this type of queue is only active while the consumer that created it is active.

Temporary queues are very often used for producers that make use of the Request / Reply MEP, where the reply messages are only relevant while the producer exists.

Hover over the to learn about the Non-Durable Temporary Queues MEP. Clicking on the icons associated with the components will lead you to further information about that component.

 

Click for a discussion on temporary queues and topic endpoints. Click for a discussion on durable queues and top endpoints. If Durable Consumer 1 stops or disconnects its Solace Session to the PS+ broker, it results in the associated Egress Queue being dissolved. The Queue2 Endpoint will continue to exist since Queue2 is a Durable Queue. There is no longer any consumer associated with the Temporary Queue1 so it is also dissolved and all references to the Persistent Event Stream are dissolved. If Durable Consumer stops or disconnects its Solace Session to the PS+ broker, it results in the associated Egress Queue being dissolved. Information on how to receive and manage Guaranteed message using APIs. Click here to get an overview of Client Egress Queue

Persistent Event Stream Maps

Persistent events are persistent. Event messages are considered to be persistent if they are placed onto non-volatile storage media after they arrive on the broker. They are suitable to application design patterns where events are required to be:

  • Processed in-order of receipt.
  • Available to consumers, even if those consumers are off-line.
  • Able to survive the loss of an event broker.

When to use persistent events

Persistent events are commonly used for MEPs where loss and duplication aren't tolerated, for example in financial transaction applications.

Separate streams for processing persistent and non-persistent events

Persistent events require more processing than non-persistent ones. Using different paths through PubSub+ brokers ensures that the processing of both persistent and non-persistent event streams through the same broker don't interfere with each other. In the following maps, we'll see the use of the Persistent Event Stream for persistent event processing.

How are persistent events processed?

There are three pieces to the puzzle, which we'll examine in detail in the upcoming sections:

Producer-Side Processing of Persistent Events

Event producers can send event messages using a guaranteed QoS by setting the message’s Persistent Delivery Mode to Persistent. This flag instructs the broker to reply to the producer with an acknowledgment when the broker has stored the message in durable storage (including HA and / or DR replication).

By hovering over the icon beside each numbered step in the map below, you can follow the processing of a persistent event through the broker.

 

1. The Producer sends an event message to the broker, and the event stream consumes and persists the message. 2. Placing a new message on the Event Stream triggers the broker to react to the change. Data Plane Tasks are alerted to determine if the message must be routed to another broker or VPN (via either DMR, MNR or VPN Bridge), and all Queue and DTE endpoints state engines are updated if they have registered interest against the newly arrived message topic. 3. Once the message is physically persisted, the Producer receives an acknowledgement. This occurs in parallel with Step 2. Click here for information about Data Plane Tasks for Durable Event Streams. Click here for information about provisioning a durable queue.

 

Data Plane Tasks for Persistent Event Streams

Data Plane Tasks operate on queue endpoints. For persistent event messaging, a queue endpoint maintains state for all messages in the Persistent Event Stream that are targeted directly to the specific queue, or are part of the queue’s topic subscriptions. There are three main functional categories associated with a queue endpoint:

  • Updating the state engine when a new message in the event stream matches the subscriptions configured against the queue.
  • Sending messages to consumer clients as either browsing or processing consumers.
  • Internal processing against state engine update events to provide internal stream processing, transaction management, redundancy, replication, queue-to-queue state-engine updates and management / administration of the state engine properties.

Updating the Queue Endpoint's state engine

The arrival of a new event message on the Persistent Event Steam causes, among other things, an update of the queue endpoint's state engine to add a pointer for the newly arrived message.

However, even if the queue endpoint has registered interest in a topic in the Persistent Event Stream, that doesn't guarantee the state engine will update the queue endpoint with a pointer to the new message. All messages must still meet the required access controls before the queue endpoint will reference the new messages.

Consuming messages

If a consumer binds a flow to the queue endpoint in order to receive messages in guaranteed delivery QoS, it still doesn't ensure the consumer will actually receive all the messages that are referenced in the queue endpoint state engine.

Guaranteed delivery QoS consumers still must be authorized at the queue level before they can consume or produce guaranteed messages that will be updated in the queue endpoint.

Guaranteed consumers may also define a selector where only a subset of the queued messages will actually be sent to the consumer. For more information about selectors, refer to Using Selectors page.

Operations the Data Plane Task performs

A newly arriving persistent message does more than update the queue endpoint record list. The Data Plane Tasks also trigger checks, and related processing operations, for the following broker features:

  • Message Replay
  • Message Replay is a Solace PubSub+ feature that allows a broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the broker.

    Some more information can be found at Message Replay Overview, and in the Replaying Messages Map section below.

  • Dead Message Queue
  • Processing against the messages in the queue endpoint, the message could be removed because of a TTL (time-to-live) setting, or a poison message redelivery setting. The message is removed from queue endpoint reference, and is automatically added to a dead message queue associated with the original queue endpoint.

    You can find more details in Dead Message Queues

  • HA Redundancy
  • The message may need to be replicated to the HA member before it's considered persisted in the primary HA broker.

    More information can be found in Redundancy & Fault Tolerance.

  • DR Replication
  • The message may need to be replicated to the DR broker cluster before it's considered persisted in the primary HA cluster.

    There are more details at Data Center Replication Overview.

  • Queue-to-Queue Multi-Phase Commit
  • If multiple queues are set with reject-msg-to-sender-on-discard, a failure to spool the same message in any one of the queues with the reject-on-discard setting, then that message is rolled back for all co-operating queues, and the producer is sent a negative acknowledgment.

    You can find more information at Reject Message To Sender on Discard.

  • Session and XA Transactions
  • The messages produced or consumed from the queue can be bracketed in either session or XA transactional semantics.

    More information can be found in Sessions and Using XA Transactions.

The Data Plane Tasks Map

By selecting the icon beside the components in the map below, you can learn more about the various Data Plane Tasks.

 

Click here for information on Message Replay Processing against the messages in the Queue Endpoint, the message could be removed because of a TTL (time-to-live) setting or “poison message” redelivery setting. The message is removed from Queue Endpoint reference, and is automatically added to a Dead Message Queue associated with the original Queue Endpoint. The message may need to be replicated to the HA mate before it's considered persisted in the primary HA broker. The message may need to be replicated to the Disaster Recovery PubSub+ broker cluster before it's considered persisted in the primary HA cluster. Selectors enable clients to specify which messages that they are interested in receiving, as determined by the messages’ header field and property values. Selectors enable clients to specify which messages that they are interested in receiving, as determined by the messages’ header field and property values. The queue owner has full unlimited permissions for the queue. That is, the owner can consume, delete, or modify topics in the queue. The queue owner has full unlimited permissions for the queue. That is, the owner can consume, delete, or modify topics in the queue. Typically, messages are published to a Queue when it's set as the destination of the message. However, you can also add a topic subscription to a Queue so that it receives any messages that are published to a matching topic destination. This section describes how to use transacted sessions to publish and/or receive a series of Guaranteed messages in a single atomic unit known as a local transaction. Local transactions only rely on a single resource (the broker) to provide messaging clients with service. This section is primarily intended for application architects, and intermediate to advanced programmers who intend to build their own XA solution rather than using the available Solace JEE Connector Architecture (JCA) Resource Adapter. When publishing guaranteed messages to a broker, messages can be discarded for reasons such as message-spool full, maximum message size exceeded, endpoint shutdown, and so on. A MessageConsumer object can be used to receive messages from a queue or for a specific topic. Client applications using the Java and .NET APIs can use the Browser interface to look at Guaranteed messages spooled for a Queue in the order of oldest to newest without consuming them. ACLs are used to control which clients may connect to which Message VPNs, and which topics clients are allowed to publish and subscribe to in their Message VPN.

 

Consumer-Side Processing of Persistent Events

Durable consumers process messages from the Persistent Event Stream by creating a flow against the queue endpoint. The Data Plane Tasks move messages to the consumer egress queue.

By selecting the icon beside each numbered step in the map below, you can follow the processing of a message to a consumer.

 

1. Both Durable Consumers receive the next Topic 3 message that is yet to be processed. 2. Both Durable Consumers acknowledge they are done processing the messages. 3. The broker is aware that there are no other durable consumers for the same topic message, so the message is removed from the Persistent Event Stream. Click here to learn how to provision a durable queue in a given Message VPN. Information on how to receive and manage Guaranteed message using APIs

Exclusive and Non-Exclusive Durable Queues

A durable queue can provide one of two types of access for consumers: Exclusive or Non-Exclusive. Both attract persistent messages even if there are no active consumers bound to the queue.

For information on how to set a durable queue's access type, you can refer to Configuring Access Types on the Configuring Queues page, but a brief summary of the two types of access is provided in the next sub-sections.

Non-Exclusive Queues

Non-exclusive queues provide load balancing and fault tolerance to durable consumers, and allow multiple consumers to bind a flow to the same queue endpoint.

Messages are delivered to consumers in a round-robin fashion to provide load balancing. If a consumer fails, its unprocessed messages are forwarded to an active consumer to ensure fault-tolerance for the consumers.

If new consumers are bound to the queue, the load is automatically distributed among the newly joined consumers against the non-exclusive queue.

Exclusive Queues

Exclusive queues provide fault tolerance to durable consumers. More than one consumer can bind a flow to the queue, but only the first bound consumer receives any messages. If the processing consumer fails, the next bound consumer will receive any unprocessed messages from the first consumer, and become the active processing consumer.

The Map

Hovering over the and icons in the map below will show you some information about exclusive and non-exclusive queue processing. By selecting the beside the various components in the map below, you can dive into some more detailed information.

 

Click here for a discussion on Exclusive Queue access type. Click here for a discussion on Non-exclusive Queue access type. Non-exclusive queues provide load balancing and fault tolerance to durable consumers. A non-exclusive queue allows multiple consumers to bind a flow to the same queue endpoint. Messages are delivered to the consumers in a round-robin fashion to allow load-balanced consumers. If a consumer fails, its unprocessed messages are forwarded to an active consumer to ensure fault tolerance for the consumers. Exclusive queues provide fault tolerance to durable consumers. More than one consumer can bind a flow to the queue, but only the first bound consumer receives any messages. If the processing consumer fails, the next bound consumer will receive any unprocessed messages from the first consumer and become the active processing consumer.

Replaying Messages Map

PubSub+ supports four different replay strategies. Which one you decide to use depends on your requirements and use-cases.

  • Message Replay
  • Message Replay is a PubSub+ feature that allows an event broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the event broker. With replay enabled, event brokers store persistent messages in a replay log.

    For more information refer to Message Replay Overview.

  • Automatic Session Redelivery
  • When PubSub+ detects a session disconnect-reconnect event, the queue endpoint will automatically replay all messages to the consumer that were in flight, or being processed by the consumer, but were not acknowledged. All replayed messages on the session have a flag indicating they were replayed due to consumer session restart.

    Information on configuring the maximum redelivery attempts for a durable queue can be found in Configuring Max Redelivery Attempts.

  • Queue Browser
  • PubSub+ supports a special consumer called a Queue Browser, which consumes messages by replaying messages from a queue, but without the queue endpoint expecting an acknowledgment, or referencing the messages as consumed.

    For more information refer to Browsing Guaranteed Messages.

  • PubSub+ Cache
  • PubSub+ Cache an external in-memory data grid for high-speed and low latency storage on non-persisted messages. Replay is by topic, and based on message depth or time.

    For more information refer to Solace PubSub+ Cache.

The Map

The following map shows you where each of the above replay strategies fits into PubSub+ event message processing. By selecting the beside the various components in the map below, you can dive into some more detailed information.

 

Message Replay is a PubSub+ feature that allows an event broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the event broker. PubSub+ Cache an external in-memory data grid for high-speed and low latency storage on non-persisted messages for purpose of replay. Replay is by topic and based on message depth or time. Click here for information on how to configure Message Replay. When PubSub+ detects a session disconnect-connect-reconnect event, the queue endpoint will automatically replay all messages to the consumer that were in flight, or being processed by the consumer, but were not acknowledged. All replayed messages on the session have a flag indicating they were replayed due to consumer session restart. PubSub+ supports a special consumer called a Queue Browser, which consumes messages by replaying messages from the queue, but without the queue endpoint expecting an acknowledgment, or referencing the messages as consumed.