Event Streaming Flows

PubSub+ separates incoming events into two separate streams depending on the Quality of Sevice (QoS) required for the events (as defined by the producer) as they are moved from producers to consumers. These Non-Persistent Event Streams and Persistent Event Streams process events using different paths. The following sections show you how that processing happens.

Non-Persistent Event Streams

Non-persistent events are ephemeral. These events provide a QoS for consumers where loss of messages is acceptable, and use a design pattern where consumers only receive messages starting from the time of connection. This allows for extremely high throughput and ultra-low latency as a trade-off against longevity. This QoS is also commonly used for Message Exchange Patterns such as Request-Reply or Publish-Subscribe in applications like market data.

Message arrival

When event messages arrive at the broker with Delivery Mode set to Direct, the message is placed at the tail of the Non-Persistent Event Stream.

Consuming messages

Non-durable consumers receive messages from the Non-Persistent Event Stream by attracting topic messages on a subscription basis. As new event messages arrive at the broker, they are placed in the consumer’s egress queue if they correspond to the client’s subscription request.

Message priority

Each Direct message has an assigned priority. The consumer’s egress queue ensures higher priority Direct messages are processed before lower priority ones. Messages of the same priority are processed in the order they arrive at the broker.

Non-persistent event processing

In the diagram below, you can follow a non-persistent event as it is processed by the PubSub+ event broker.

The detailed steps of this process flow (corresponding to the numbers in the diagram) are as follows:

  1. Message producers send events with the header's Persistent Delivery Mode set to Direct. The messages are written to the tail of the Non-Persistent Event Stream based on the order they arrive at the broker. Suppose a message is sent with the Persistent Delivery Mode set to Persistent. In that case, it will also be added to the Non-Persistent Event Stream for those consumers who are subscribed to the topic but don't require message persistence.
  2. The arrival of messages in the Non-Persistent Event Stream causes the Data Plane to determine which Non-Durable Consumers are interested in the message.
  3. A reference to the message is moved to the appropriate Consumer Egress Queues for all consumers whose topic subscriptions match the message's topic. Based on the priority of the messages, the messages are then queued in TCP for delivery to the appropriate consumer. Refer to Client Egress Queue Structure Overview for more information.
  4. Once all consumers that have registered interest in the same topic message, and their Egress Queue schedulers have moved the message to the TCP output queue, the message is deleted from the Non-Persistent Event Stream. See Direct Messages for more information.

To find out more about the components that are involved, see the following:

  • Non-persistent Producer: Further discussion on publishing Direct messages.
  • Topics: Understand the role played by topics in the message flow from producers to consumers.
  • Client Egress Queue: A description of how the egress queue manages messages of different priorities.
  • Non-Durable Consumer: Learn more about receiving and managing Direct messages using messaging APIs.

Data Plane Tasks for Incoming TCP Events

All messages enter a Solace PubSub+ broker via custom ingress TCP buffers. The arrival of new TCP event messages cause the Data Plane to move the TCP data to the Non-Persistent Event Stream for further processing. However, it should be noted that messages are only moved to the event stream if they pass ingress ACL processing.

Messages in the Non-Persistent Event Stream are ephemeral, and message discard is acceptable. All messages, including messages that are flagged with a Persistent Delivery Mode set to Persistent or Direct are first placed into the Non-Persistent Event Stream when they arrive over TCP/IP. The Data Plane immediately moves messages flagged as Direct to the consumer's egress queue where the consumer has a matching topic subscription.

Messages flagged as Persistent are immediately placed into the Persistent Event Stream. If the persisted mode messages are discarded before they are processed into the Persistent Event Stream, the producer API ensures redelivery to the broker, and the Persisted Event Stream will automatically discard redelivered duplicates.

Operations the Data Plane Performs on Messages Arriving over TCP

Once the messages pass the ingress ACL check, they are placed in the Non-Persistent Event Stream, and are further processed by the Data Plane, as follows:

  • Promotion

    • Some messages will have arrived with the Persistent Delivery Mode set to Direct, but a queue may have a topic subscription for those messages. Topic messages subscribed to from queues are promoted to Persistent, and moved to the Persistent Event Stream, but no ACK is sent to the producer.

    For more information about promotion, see Message Promotion and Demotion.

  • Demotion

    • Some messages will arrive in the Non-Persistent Event Queue that have the Persistent Delivery Mode set to Persistent. However, there may be topic subscribers who want the same message while it's being moved to the Persistent Event Stream. This message is demoted to Direct for delivery directly to the consumer egress queue for ultra-low latency and high throughput processing for consumers who don't require guaranteed delivery.

    Further information about demotion can also be found at Message Promotion and Demotion.

  • Prioritization

    • Based on an indication from the producer, the message is placed into one of three consumer egress queue priority streams.

    For more information refer to Client Egress Queue Structure Overview.

  • Shared Subscriptions

    • Messages can be delivered in a round-robin fashion to a group of topic consumers to provide load balancing similar to non-exclusive queue processing.

    More information can be found at Shared Subscriptions.

  • Persisted Message moved to Persistent Event Stream

    • All messages from the TCP network are placed directly into the Non-Persistent Event Stream. If the message's Persistent Delivery Mode is set to Persistent, the message is moved and persisted against the Persistent Event Stream.

  • Logging

    • Event messages are logged in real-time based on multiple conditions ranging from threshold settings to messages arriving without any subscribers. This is a log of message activity, not tracing of the full message.

    Information about configuring logging for various APIs can be found in Configuring Logging.

  • Subscription Binding

    • When consumers indicate they want specific topic messages for Direct mode delivery (including demoted messages), the attracted topic messages are now referenced in the consumer’s egress queue for processing to the TCP outbound buffers.

  • Consumer Egress Queue

    • A reference to the messages in the Non-Persistent Event Queue that are scheduled for TCP delivery to specific consumers. The reference will only be added to the egress queue if the consumer’s security ACL profile allows the delivery.

    For more information refer to Client Egress Queue Structure Overview.

Data Plane Tasks

The following diagram illustrates the tasks performed by the Data Plane:

The following discussion and links provide information about the components illustrated in the diagram.

  • Consumer Egress Queue: Learn more about egress per-client priority queues on brokers, and the commands that you can use to configure them.
  • Egress Topic ACL: Learn more about egress topic ACLs.
  • Ingress Topic ACL: You can use ACLs to control the topics to which clients are allowed to publish.
  • Promotion: Message promotion is the situation where a producer sends Direct messages, and the consumer receives these message from a Guaranteed messaging endpoint.
  • Demotion: Message demotion is the situation where the producer sends Persistent messages, and there are consumers that want to receive these messages, but can tolerate lost messages.
  • Prioritization: When you enable an endpoint to respect message priority, the priority field in messages from producers are respected for all guaranteed and promoted direct messages.
  • Shared Subscriptions: Shared subscriptions can be used to load balance large volumes of client data across multiple instances of back end data center applications.
  • Persistent Message moved to Persisted Event Stream: All messages from the TCP network are placed into the Non-Persistent Event Stream directly. Based on the Delivery Mode in the message set to Persistent, the message is moved and persisted against the Persistent Event Stream.
  • Logging: Learn how to use messaging APIs for logging.
  • Subscription Binding: Typically, messages are published to a Queue when it's set as the destination of the message. However, you can also add a topic subscription to a Queue so that it receives any messages that are published to a matching topic destination.

Non-Durable Temporary Queues

There are several message exchange patterns (MEP)s where there's no requirement for a persistent queue to exist beyond the life of a durable consumer. These queues are called temporary queues; this type of queue is active only while the consumer that created it is active.

Temporary queues are very often used for producers that make use of the Request / Reply MEP, where the reply messages are only relevant while the producer exists.

In the following diagram, you can follow a message as it's processed on a Non-Durable Temporary Queue:

The detailed steps of this process flow (corresponding to the numbers in the diagram) are as follows:

  1. If Durable Consumer 1 stops or disconnects its Solace Session to the event broker, it results in the associated Egress Queue being dissolved. The Queue 2 Endpoint will continue to exist since Queue 2 is a Durable Queue.
  1. If Durable Consumer 2 stops or disconnects its Solace Session to the event broker, it results in the associated Egress Queue being dissolved.
  2. There is no longer any consumer associated with the Temporary Queue 1 so it is also dissolved and all references to the Persistent Event Stream are dissolved.

The following links provide information about the components illustrated in the above diagram.

Persistent Event Streams

Event messages are considered to be persistent if they are placed onto non-volatile storage media after they arrive on the broker. Persistent messages are suitable for application design patterns where the event messages must be:

  • Processed in-order of receipt.
  • Available to consumers, even if those consumers are off-line.
  • Able to survive the loss of an event broker.

When to use persistent events

Persistent events are commonly used for MEPs where loss and duplication aren't tolerated, for example in financial transaction applications.

Separate streams for processing persistent and non-persistent events

Persistent events require more processing than non-persistent ones. PubSub+ event brokers use different paths for these two message types to ensure that the respective processes don't interfere with each other.

How are persistent events processed?

Persistent event messages are processed in three main phases that are discussed in detail in later sections:

Producer-Side Processing of Persistent Events

Event producers can send event messages using a guaranteed QoS by setting the message's Persistent Delivery Mode to Persistent. This flag instructs the broker to reply to the producer with an acknowledgment when the broker has stored the message in durable storage (including HA and / or DR replication).

Persistent event processing

In the diagram below, you can follow a persistent event as it is processed by the PubSub+ event broker.

The detailed steps of this process flow (corresponding to the numbers in the diagram) are as follows:

  1. The Producer sends an event message to the event broker, and the event stream consumes and persists the message. For more information, see Publishing Guaranteed Messages.
  2. Placing a new message on the Event Stream triggers the broker to react to the change. The Data Plane is alerted to determine if the message must be routed to another broker or VPN (via either DMR, MNR, or VPN Bridge). All Queue and DTE endpoints state engines are updated if they have registered interest against the newly arrived message topic. Take a look at Guaranteed Messaging Operation to learn more.
  3. Once the message is physically persisted, the Producer receives an acknowledgment. This occurs in parallel with Step 2. More information is available in Guaranteed Messaging Operation.

To find out more about provisioning a durable queue, see Configuring Queues.

Data Plane Tasks for Persistent Event Streams

The Data Plane operates on queue endpoints. For persistent event messaging, a queue endpoint maintains state for all messages in the Persistent Event Stream that are targeted directly to the specific queue, or are part of the queue's topic subscriptions. There are three main functional categories associated with a queue endpoint:

  • Updating the state engine when a new message in the event stream matches the subscriptions configured against the queue.
  • Sending messages to clients as either browsing or processing consumers.
  • Internal processing against updates to the state engine to provide internal stream processing, transaction management, redundancy, replication, queue-to-queue state-engine updates and management / administration of the state engine properties.

Updating the Queue Endpoint's state engine

The arrival of a new event message on the Persistent Event Steam causes, among other things, an update of the queue endpoint's state engine to add a pointer for the newly arrived message.

However, even if the queue endpoint has registered interest in a topic in the Persistent Event Stream, that doesn't guarantee the state engine will update the queue endpoint with a pointer to the new message. All messages must still meet the required access controls before the queue endpoint will reference the new messages.

Consuming messages

If a consumer binds a flow to the queue endpoint in order to receive messages in guaranteed delivery QoS, it still doesn't ensure the consumer will actually receive all the messages that are referenced in the queue endpoint state engine.

Guaranteed delivery QoS consumers still must be authorized at the queue level before they can consume or produce guaranteed messages that will be updated in the queue endpoint.

Guaranteed consumers may also define a selector where only a subset of the queued messages will actually be sent to the consumer. For more information about selectors, refer to Using Selectors page.

Operations the Data Plane performs

A newly arriving persistent message does more than update the queue endpoint record list. The Data Plane also triggers checks and related processing operations for the following broker features:

  • Message Replay

    Message Replay is a Solace PubSub+ feature that allows a broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the broker.

    More information can be found at Message Replay and in the Replaying Messages section below.

  • Dead Message Queue

    Processing against the messages in the queue endpoint, the message could be removed because of a TTL (time-to-live) setting, or a poison message redelivery setting. The message is removed from queue endpoint reference, and is automatically added to a dead message queue associated with the original queue endpoint.

    You can find more details in Configuring Dead Message Queues.

  • HA Redundancy

    The message may need to be replicated to the HA member before it's considered persisted in the primary HA broker.

    More information can be found in Event Broker Redundancy for High Availability.

  • DR Replication

    The message may need to be replicated to the DR broker cluster before it's considered persisted in the primary HA cluster.

    There are more details at Data Center Replication for Disaster Recovery.

  • Queue-to-Queue Multi-Phase Commit

    If multiple queues are set with reject-msg-to-sender-on-discard, a failure to spool the same message in any one of the queues with the reject-on-discard setting, then that message is rolled back for all co-operating queues, and the producer is sent a negative acknowledgment.

    You can find more information at Reject Message To Sender on Discard.

  • Session and XA Transactions

    The messages produced or consumed from the queue can be bracketed in either session or XA transactional semantics.

    More information can be found in Sessions and Using XA Transactions.

Data Plane Tasks

The diagram below illustrates various tasks performed by the Data Plane:

The components shown in the diagram are as follows:

  • Stream ACL: ACLs are used to control which clients may connect to which Message VPNs, and which topics clients are allowed to publish and subscribe to in their Message VPN.
  • Selector: Selectors enable clients to specify which messages that they are interested in receiving, as determined by the messages’ header field and property values.
  • Queue: The queue owner has full unlimited permissions for the queue. That is, the owner can consume, delete, or modify topics in the queue.
  • Queue Permissions: The queue owner has full unlimited permissions for the queue. That is, the owner can consume, delete, or modify topics in the queue.
  • Queue Consumer: A MessageConsumer object can be used to receive messages from a queue or for a specific topic.
  • Queue Browser: Client applications using the Java and .NET APIs can use the Browser interface to look at Guaranteed messages spooled for a Queue in the oldest to newest order without consuming them.
  • Message Replay: Get an overview information on Message Replay.
  • Dead Message Queue: Processing against the Queue Endpoint messages, the message could be removed because of a TTL (time-to-live) setting or "poison message" redelivery setting. The message is removed from Queue Endpoint reference and is automatically added to a Dead Message Queue associated with the original Queue Endpoint.
  • HA Redundancy: The message may need to be replicated to the HA mate before it's considered persisted in the primary HA broker.
  • DR Replication: The message may need to be replicated to the Disaster Recovery PubSub+ broker cluster before it's considered persisted in the primary HA cluster.
  • Queue-to-Queue Multi-phase Commit Replication: When publishing guaranteed messages to a broker, messages can be discarded for reasons such as message-spool full, maximum message size exceeded, endpoint shutdown, and so on.
  • Session Transaction: This section describes how to use transacted sessions to publish and/or receive a series of Guaranteed messages in a single atomic unit known as a local transaction. Local transactions only rely on a single resource (the broker) to provide messaging clients with service.
  • XA Transaction: This section is primarily intended for application architects, and intermediate to advanced programmers who intend to build their own XA solution rather than using the available Solace JEE Connector Architecture (JCA) Resource Adapter.
  • Subscription Binding: Typically, messages are published to a Queue when it's set as the destination of the message. However, you can also add a topic subscription to a Queue so that it receives any messages that are published to a matching topic destination.

Consumer-Side Processing of Persistent Events

Durable consumers process messages from the Persistent Event Stream by creating a flow against the queue endpoint. The Data Plane moves messages to the consumer egress queue.

In the diagram below, you can follow the processing of a message to a consumer.

The detailed steps of this process flow (corresponding to the numbers in the diagram) are as follows:

  1. Both Durable Consumers receive the next Topic 3 message that is yet to be processed. Learn more at Receiving Guaranteed Messages.
  2. Both Durable Consumers acknowledge they are done processing the message. Learn more at Acknowledging Messages Received by Clients.
  3. The broker is aware that there are no other durable consumers for the same topic message, so the message is removed from the Persistent Event Stream. Additional information available at Acknowledging Messages Received by Clients.

To find out more about the components that are involved, see the following:

  • Queue: Get information on provisioning a durable queue.
  • Durable Consumer: Information on how to receive and manage Guaranteed message using APIs

Exclusive and Non-Exclusive Durable Queues

A durable queue can provide one of two types of access for consumers: Exclusive or Non-Exclusive. Both attract persistent messages even if there are no active consumers bound to the queue.

For information on how to set a durable queue's access type, you can refer to Configuring Access Types on the Configuring Queues page, but a brief summary of the two types of access is provided in the next sub-sections.

Non-Exclusive Queues

Non-exclusive queues provide load balancing and fault tolerance to durable consumers, and allow multiple consumers to bind a flow to the same queue endpoint.

Messages are delivered to consumers in a round-robin fashion to provide load balancing. If a consumer fails, its unprocessed messages are forwarded to an active consumer to ensure fault-tolerance for the consumers.

If new consumers are bound to the queue, the load is automatically distributed among the newly joined consumers against the non-exclusive queue.

Exclusive Queues

Exclusive queues provide fault tolerance to durable consumers. More than one consumer can bind a flow to the queue, but only the first bound consumer receives any messages. If the processing consumer fails, the next bound consumer will receive any unprocessed messages from the first consumer, and become the active processing consumer.

Exclusive and Non-Exclusive Durable Queue Process Flow

The diagram below provides overview information about exclusive and non-exclusive queues.

The detailed steps of this process flow (corresponding to the letters in the diagram) are as follows:

  1. Non-exclusive queues provide load balancing and fault tolerance to durable consumers. A non-exclusive queue allows multiple consumers to bind a flow to the same queue endpoint. Messages are delivered to the consumers in a round-robin fashion to allow load-balanced consumers. If a consumer fails, its unprocessed messages are forwarded to an active consumer to ensure fault tolerance for the consumers.
  2. Exclusive queues provide fault tolerance to durable consumers. More than one consumer can bind a flow to the queue, but only the first bound consumer receives any messages. If the processing consumer fails, the next bound consumer will receive any unprocessed messages from the first consumer and become the active processing consumer.

To find out more about the components illustrated above, see Queue Access Types.

Replaying Messages

PubSub+ supports four different replay strategies. Which one you decide to use depends on your requirements and use-cases.

  • Message Replay

    Message Replay is a PubSub+ feature that allows an event broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the event broker. With replay enabled, event brokers store persistent messages in a replay log.

    For more information refer to Message Replay.

  • Automatic Session Redelivery

    When PubSub+ detects a session disconnect-reconnect event, the queue endpoint will automatically replay all messages to the consumer that were in flight, or being processed by the consumer, but were not acknowledged. All replayed messages on the session have a flag indicating they were replayed due to consumer session restart.

    Information on configuring the maximum redelivery attempts for a durable queue can be found in Configuring Max Redelivery Attempts.

  • Queue Browser

    PubSub+ supports a special consumer called a Queue Browser, which consumes messages by replaying messages from a queue, but without the queue endpoint expecting an acknowledgment, or referencing the messages as consumed.

    For more information refer to Browsing Guaranteed Messages.

  • PubSub+ Cache

    PubSub+ Cache an external in-memory data grid for high-speed and low latency storage on non-persisted messages. Replay is by topic, and based on message depth or time.

    For more information refer to PubSub+ Cache.

Message Replay Process Flow

The following diagram shows you where each of the above replay strategies fits into PubSub+ event message processing.

The following discussion and links provide information about the components illustrated in the diagram.

  • PubSub+ Cache: An external in-memory data grid for high-speed and low latency storage on non-persisted messages for purpose of replay. Replay is by topic and based on message depth or time.
  • Message Replay: A PubSub+ feature that allows an event broker to resend messages to new or existing clients that request them, hours or even days after those messages were first received by the event broker.
  • Replay Log: Learn how to configure Message Replay.
  • Automatic Session Redelivery: When PubSub+ detects a session disconnect-connect-reconnect event, the queue endpoint will automatically replay all messages to the consumer that were in flight or being processed by the consumer, but were not acknowledged. All replayed messages on the session have a flag indicating they were replayed due to consumer session restart.
  • Queue Browser: PubSub+ supports a special consumer called a Queue Browser, which consumes messages by replaying messages from the queue, but without the queue endpoint expecting an acknowledgment, or referencing the messages as consumed.