Direct Messages

Direct messaging provides reliable, but not guaranteed, delivery of messages from the Solace message bus to consuming clients. It is the default message delivery system for PubSub+. It doesn't require any additional configuration beyond that required to set up and start the event broker. By default, direct messaging is always available to all clients that connect to an event broker.

Direct Messaging Characteristics

Direct messages have the following characteristics:

  • They are delivered to subscribing clients in the order that publishers publish them.

  • They don't require acknowledgment of receipt by subscribing clients.

  • They aren't spooled on the message bus for consuming clients.

  • They aren't retained for a client when the client isn't connected to the event broker.

  • They can be discarded before delivery in some situations:

    • a client disconnects from an event broker while messages are being published

    • a client fails to consume messages at the published rate and an egress message buffer overflow results

    • congestion or system failures.

Direct Messaging Use Cases

Direct messaging is useful in the following situations:

  • Extremely high message rates and extremely low latency are required.

  • Consuming clients can tolerate message loss in the event of network congestion.

  • Messages don't need to be persisted for later delivery to slow or offline consumers.

  • Messages must be efficiently published to a large number of clients with matching subscriptions.

For more information on features available with Direct messaging, see the following sections:

Shared Subscriptions

Shared subscriptions can be used with Direct messaging to load balance large volumes of client data across multiple instances of backend data center applications. They are especially useful in situations where those applications parallelize the processing of published messages. Shared subscriptions are not supported with Guaranteed messaging.

To control whether a client can create or join in a shared subscription, the allow-shared-subscriptions attribute in the client’s client-profile is set to the desired value. For more information, see Allowing Shared Subscriptions.

Syntax of Shared Subscriptions

A shared subscription has the form:

#share/<ShareName>/<topicFilter>

Where:

  • ShareName is the identifier associated with the shared subscription. It may not contain wildcards (* or >).
  • topicFilter is the topic filter.

Shared subscriptions are exported (advertised by the local node to other nodes in the network) by default. To prevent this export, start the shared subscription with #noexport, as follows:

#noexport/#share/<ShareName>/<topicFilter>

Where:

  • ShareName is the identifier associated with the shared subscription. It may not contain wildcards (* or >).
  • topicFilter is the topic filter.

For more information, see Preventing Subscription Export and the #noexport entry of the table in Reserved Topics.

How Shared Subscriptions Work

To illustrate how shared subscriptions work, consider the following example:

  • #share/ottawa/weathersensor/yow/temperature

A number of clients are subscribing to #share/ottawa/weathersensor/yow/temperature. When a message is published to weathersensor/yow/temperature one of those clients is randomly chosen to receive the message. For each message that arrives that matches weathersensor/yow/temperature, a new random choice determines which subscribing client receives the message. With a large enough number of messages, n, to subscribing clients, m, the number of messages delivered to each subscriber evens out to approximately n/m.

Shared Subscription Identifiers

A ShareName is included in a shared subscription so there can be more than one shared subscription on an event broker that has the same topicFilter.

For example, consider the following shared subscriptions, which differ only by ShareName:

  • #share/weatherapp/weathersensor/*/temperature
  • #share/temperaturelog/weathersensor/*/temperature

Although the shared subscriptions have the same topicFilter, they are unique. When a message is published to weathersensor/yow/temperature, and a number of clients are subscribing to #share/weatherapp/weathersensor/*/temperature, one of those clients is randomly chosen to receive the message. Likewise, from all the clients subscribing to #share/temperaturelog/weathersensor/*/temperature, one of the clients with that subscription is randomly chosen to receive the message.

In typical use, application instances that share messages on a subscription agree on a ShareName to use in the shared subscription. Messages with topics matching the shared subscription are then randomly distributed between those application instances.

Feature Interactions

Guaranteed Messaging

Shared subscriptions are not allowed on queues and will be rejected, whether they are configured or signaled in.

Eliding

When a client, whose client-profile indicates eliding is being used, adds a shared subscription, eliding won't be performed on messages matching the shared subscription.

Class-of-Service

Direct Messaging Class of Service (CoS) is always handled as COS1 when being delivered on a shared subscription, regardless of the message’s CoS.

No Local Sessions

Shared subscriptions ignore the no-local attribute of a session. If a client adds a shared subscription on a no-local session, messages published on that session are eligible for delivery on the same session via the shared subscription.

API LocalDispatchOnly subscriptions

Shared subscriptions cannot be added as API LocalDispatchOnly subscriptions. If an application is trying to create a shared subscription limited in scope to a single session, use some randomly generated content within the shared subscription’s sharedSubID that would be unknown to other applications.

PubSub+ Cache

Shared subscriptions are not supported as being configurable against PubSub+ Cache.

Dynamic Message Routing (DMR) and Multi-Node Routing (MNR)

Shared subscription topic filters are exported. This attracts traffic that matches the shared subscription from other nodes, but it doesn't have the effect of a distributed shared subscription. If a remote node has the same shared subscription, they are treated as different shared subscriptions, where one member of the group on each node will receive a copy of a published message that matches the shared subscription.

Message Eliding

Message eliding allows client applications to receive Direct messages published to topics that they subscribe to, at a rate they can manage, rather than queue outdated messages. Message eliding can be useful in situations where messages are consumed at a slower rate then they are published due to slow consumers or human interaction.

When using message eliding, some message loss must be acceptable. For example, a producer may publish multiple messages per second that provide current state data, such as weather, a vehicle location, or stock market information, and the consumer cares only about the current state.

Only Direct messages can be elided. Messages received through shared subscriptions can't be elided.

To use message eliding:

  • Publishing client applications must flag published message as eligible for message eliding.

    • Although only a Direct message can be elided, it is possible for a publishing client to flag Guaranteed messages as eliding‑eligible. However, the message is not elided unless the delivery mode of the message is changed to Direct, which can happen if the message is published to a topic that matches a client’s topic subscription.

    • Messages published by MQTT client applications are treated as non‑eliding eligible.

  • Receiving client applications must be assigned a client profile through its client username that permits it to:

    • Use message eliding

    • After the first message, receive subsequent messages with a time delay interval. The delay interval, which is configured in the client profile, controls the rate of message updates sent to a client on a topic-by-topic basis (for example, five messages per second per topic).

    • Receive up to the maximum number of topics the event broker can track for performing the eliding function on each client connection (up to 32,000 per client as configured in the client profile, with a total of 2,000,000 per event broker; the default is 256 per client). Once the maximum number of topics is reached, the event broker ages out the elided topics for the client to prevent the consumption of more eliding resources than have been allocated for the connection. Eliding behavior then continues as if this were a new client connection, and a one-time Syslog event is generated on a per client basis.

      • Event brokers dynamically track the number of topics on a client connection. Whenever this number is below the set maximum, eliding is applied to the new incoming messages.

      • For subscribers with wildcard subscriptions, each topic that matches the subscription is elided, up to the maximum number of subscriptions specified by the client profile.

      • Solace recommends that consuming clients don't use discard indications when using message eliding. In a situation where an event broker’s egress priority queue for a client fills up with received messages, the oldest messages on the egress queue are discarded to make room for newly arriving messages, and the message at the head of the queue is flagged with the discard indication. However, if eliding is enabled, that message could be elided, and the client would not receive the discard indication.

Message Eliding Example

The following diagram and example explain how message eliding works:

Diagram showing the process described in the surrounding text.

  1. The Producer publishes messages to a topic at a rate of one message per millisecond (ms). Each message is flagged as eliding-eligible.

  2. The Consumer is assigned a client profile that has message eliding enabled, and has an eliding delay of 200 ms, which means it receives 5 messages per second..

  3. When the first message, M1, for the topic arrives at the event broker, it's sent to the Consumer without any delay.

  4. When the second message, M2, for the topic arrives at the event broker a millisecond later, it's held by the event broker and not sent to the Consumer because message M1 for the same topic was sent to the Consumer within the 200ms delay time.

  5. When subsequent messages, M3, M4, .... Mn, for the same topic arrive at the event broker for the Consumer, each new message replaces the previous message and is continuously held by the event broker.

  6. Following the 200ms delay after message M1 was sent to the Consumer, the currently held message is sent to the Consumer.

Message Eliding Use Cases

Use cases for message eliding feature include:

Eliding for congestion management
A client application wants to receive every message when it is able to keep up with the message flow. If the client can't keep up, then any queued messages are elided to send only the most recent message for each topic. For this use case the delay interval time is 0.
Eliding for message rate control
A client application wants to receive no more than five messages per second per topic. In this case, the event broker rate controls the output of messages to the client. For this use case the delay interval time is 200 ms (five messages per second per topic).

Example uses include:

Streaming market data to human traders
Even though market data updates might be published at a very high rate, humans can only deal with a few updates per second. The client always wants the latest information, but at a slower rate than the total feed rate. Message eliding can be used to limit the output stream to a few updates per topic per second.
Controlling update rates to subscribers over a Wide Area Network (WAN)
Sometimes, WAN bandwidth versus receiver processing capacity requires that only a subset of the entire feed rate be provided over the WAN. Message eliding can be used to control the message update rates to a client.

Client Subscription Management

When consuming clients are authenticated on the event broker, they can add and remove topic subscriptions for Direct messages published to the Message VPN they are connected to. These topic subscriptions are not durable - the client subscriptions are not maintained on the event broker after the consumer disconnects from the event broker.

A client typically performs the following steps for managing their own subscriptions:

  1. The client connects to the event broker and authenticates itself. At this point, the event broker has no subscriptions for that client.
  2. The client adds subscriptions to the event broker.
  3. The client receives messages that match the requested subscriptions.
  4. When the client disconnects from the event broker, the event broker immediately removes all subscriptions associated with that client.

The topics that a client is permitted to subscribe to can be limited by an ACL assigned to the client username account used by the client. For more information, see Controlling Client Access with ACL Profiles.

Managing Subscriptions on Behalf of Other Clients

A client application can manage subscriptions on behalf of other clients within a Message VPN when its client username is configured to be a subscription manager (see Configuring Subscription Managers ). This is useful for centralizing the assignment of client applications and services to direct messaging subscriptions. A client acting as a subscription manager has no control over Guaranteed messaging.

When a client is configured as a subscription manager, the subscriptions that it manages are subject to the access control permissions configured on that client’s associated Access Control List (ACL) profile - the ACLs of destination clients are not used. So, the client configured as a subscription manager can't add or remove subscriptions that its own ACL rules would deny adding. This restriction prevents the client from inadvertently adding or removing subscriptions that it's not entitled to.

When a client is configured as a subscription manager, other clients within the Message VPN typically perform the following steps for managing their subscriptions:

  1. The client connects to the event broker.
  2. The client notifies the subscription manager that it's ready to receive messages.

  3. The subscription manager authenticates the client, determines the subscription set needed by the client, and adds subscriptions to the event broker on behalf of the client. Subscriptions added by the subscription manager on behalf of the client have the same subscription rate (subscriptions per second) as those added by the client directly for itself.

  4. The client receives messages that match the subscription set.
  5. Once the client is done, it disconnects and its associated subscription set is deleted from the event broker.

Once subscriptions have been added by a subscription manager on behalf of another client, they behave like any other subscriptions (for example, the subscriptions are removed if the client is disconnected). Disconnecting the subscription manager has no effect on the subscriptions already added by it.

For information on how clients using Solace APIs can act as subscription managers to add and remove subscriptions on behalf of others, see Managing Topic Subscriptions on Behalf of Other Clients.