Using MQTT

The Message Queuing Telemetry Transport (MQTT) protocol is a lightweight, open protocol that can be used for Machine to Machine (M2M) and Internet of Things (IoT) use cases.

Solace PubSub+, as of version 7.1.1, supports OASIS Standard MQTT v3.1.1. This allows client applications to inter-operate with the Solace PubSub+ without relying on Solace-specific APIs or custom software development.

Like the Solace Message Format (SMF) protocol, MQTT is a topic-based client/server, publish/subscribe protocol that enables client applications to exchange messages without direct knowledge of how those messages will be delivered to other interested clients.

Clients can publish messages to defined topics, and they may use topic filters (that is, topic subscriptions) to receive messages that are published to those matching topics. All topics are UTF-8 strings, which consist of one or more topic levels that are separated by forward slash “/” characters. Separating topic levels using slashes creates a hierarchy of information for organizing topics.

The SMF and MQTT protocols use similar topic syntax, therefore SMF and MQTT messaging applications can be used together in the same messaging network. That is, when an intersecting topic hierarchy is used, MQTT clients can receive messages published by non-MQTT clients (for example, SMF or REST clients), and non-MQTT clients can receive messages published by MQTT clients. However, there are some differences in topic syntax and usage between SMF and MQTT that must be considered before deploying MQTT applications in a network alongside SMF applications. For more information, see Topic Support & Syntax.

MQTT clients connect to an MQTT server (in this case, an event broker), which is responsible for maintaining their subscription sets and directing published messages to clients that have matching topic filters. Topic filters function the same as topic subscriptions in the Solace PubSub+, so for consistency, they will be referred to as topic subscriptions throughout this document.

An MQTT client connection to a specific Message VPN on an event broker is made through a dedicated MQTT port configured for the Message VPN that they are connecting to. The MQTT client connection also requires an MQTT session on the server. An MQTT session is represented as a managed object in Solace PubSub+, and it contains the session state for the MQTT client (that is, its subscriptions and messages).

  • Solace's implementation of MQTT complies with OASIS Standard MQTT v3.1.1.1. There is an annotated version of the specification (MQTT Version 3.1.1) that highlights any deviations, limitations, or choices made in the “SHOULD” and “MAY” clauses of the protocol specification for the Solace implementation. It is strongly recommended that network architects and programmers review this document.
  • MQTT is not supported on the Solace PubSub+ 3230 or Solace PubSub+ appliances that use a Network Acceleration Blade (NAB) model NAB-0401EM.

MQTT Component Concept Map

If you haven't used MQTT and Solace PubSub+ together before, you might want to look at the MQTT section of the Overview: How Apps Interact with PubSub+ Messaging Components page. It'll take you on a tour of the PubSub+ event broker's messaging components, and explain at a high level how they make data move from producers to the broker, and from the broker to consumers, with MQTT.

MQTT Sessions

An MQTT session object is a virtual representation of an MQTT client connection that exists as a managed object on an event broker. An MQTT session holds the state of an MQTT client (that is, it is used to contain a client’s QoS 0 and QoS 1 subscription sets and any undelivered QoS 1 messages).

An MQTT session can be created:

  • Automatically when a client successfully connects to the event broker.
  • By an administrator using the Solace CLI, SEMP, or SolAdmin. Although the MQTT specification does not require administrator‑provisioned MQTT sessions to be supported, they are allowed, and they provide more flexibility for application development.

:  MQTT sessions should not be confused with Solace sessions (that is, non‑MQTT sessions).

MQTT Session Persistence

When a connecting client provides a CLEAN=1 flag for the MQTT session, the client’s MQTT session and any related information (subscription sets and undelivered messages) are not persisted after the client disconnects. That is, the flag ensures that the session “cleans up” after itself and no information is stored on the event broker after the client disconnects. This is true even if the session was administratively provisioned (through CLI or SEMP). If the client provides a CLEAN=0 flag, the MQTT session is persisted on the event broker, which means that the client’s client ID, topic subscriptions, QoS levels, and undelivered messages are all maintained (that is, they are not cleaned up). The client may then reconnect to the persisted session later.

An MQTT session can be deleted:

  • automatically when a client that created the MQTT session with a CLEAN=1 flag disconnects
  • when a client creates a new MQTT session with a CLEAN=1 flag and the same session identifier as the previous session
  • manually by administrators through the Solace CLI, SEMP, or SolAdmin

Quality of Service Levels

MQTT publish messages and topic subscriptions are assigned separate quality of service (QoS) levels, which determine the level of guarantee applied to message delivery. The MQTT QoS levels are:

  • QoS 0—At most once delivery. No response is required by the receiver (whether it is an event broker or a subscriber), and no retry attempts are made if the message is not delivered.
  • The Solace equivalent to QoS 0 is a message delivery mode of Direct.

  • QoS 1—At least once delivery. This level ensures that the message is delivered at least once. In a QoS 1 exchange, the receiver (whether it is an event broker or a subscriber) is required to send an acknowledgment of the message to indicate that it has been received.
  • The Solace equivalent to QoS 1 is a message delivery mode of Guaranteed (that is, non‑persistent or persistent). For more information, see Guaranteed Messages.

  • An event broker requires a Guaranteed messaging configuration to provide QoS 1 service. If an event broker is not configured for Guaranteed messaging, all QoS 1 and QoS 2 MQTT topic subscriptions will be downgraded to QoS 0. Additionally, QoS 1 and QoS 2 messages will be accepted by the event broker, but they will be delivered as QoS 0 messages. For more information on how to configure an event broker for Guaranteed Messaging, see Guaranteed Messaging.
  • QoS 2—Exactly once delivery. Solace converts published QoS 2 messages to QoS 1.

The following figure shows how different QoS levels may be applied to the same message. From a publisher to an event broker, an MQTT publish message uses the QoS level assigned by the message publisher. From the event broker to the message subscriber, an MQTT publish message uses the QoS level set by the matching topic subscription for the consuming client.

QoS Levels Applied During Message Delivery
QoS Levels Applied During Message Delivery

When there is a topic subscription match for an MQTT publish message, a consuming client will receive the message with the lowest possible QoS level for its subscription. For example, if a message is published with a QoS of 1, and it matches the client’s QoS 0 subscription, it will be delivered with a QoS of 0. However, because MQTT uses a one-to-many pub/sub messaging model, that message could also be delivered to a matching QoS 1 topic subscription with a QoS of 1.

MQTT Topics

As a publish/subscribe messaging protocol, MQTT relies on a hierarchical topics. Clients can publish messages on specific topics, and clients can receive published messages that match their current topic subscriptions.

MQTT topics are UTF-8 strings, which consist of one or more topic levels that are separated by forward slash “/” characters. Separating topic levels using slashes creates a hierarchy of information for organizing topics.

Connected clients can publish MQTT messages to defined topics, and they may use topic filters (that is, topic subscriptions) to receive messages that are published to those matching topics.

Payload Handling When Message Types Change

This section discusses how the payloads of MQTT publish messages received by the event broker (that is, ingress messages) are handled when they are subsequently delivered as different message types (either as SMF or REST messages) to non‑MQTT clients that have matching topic subscriptions. It also discusses how the message payloads of received SMF or REST message are handled when they are subsequently delivered as MQTT publish messages to consuming MQTT clients with matching topic subscriptions.

MQTT Ingress, SMF Egress

When an event broker receives an MQTT publish message from a client, the received message’s payload, which is a sequence of bytes, is encapsulated in the resulting egress SMF message as a binary attachment.

MQTT Ingress, REST Egress

When an event broker receives an MQTT publish message from a client, the received message’s payload is delivered with a Content-Type of application/octet‑stream.

SMF Ingress, MQTT Egress

An SMF message may contain binary data and XML data payloads, and in some cases, user-defined and Solace-defined message header fields.

When an event broker receives an SMF publish message from a client for which there is a matching MQTT topic subscription, the payload of the message is processed before it is sent to the MQTT client as an MQTT publish message.

The SMF message’s XML message payload or binary attachment can be used for the payload for the MQTT publish message, but not both.

  • If the SMF message contains only a binary attachment, the following occurs:
    • If there is no binary metadata, then the binary attachment is copied into the payload field of the MQTT publish message.
    • If there is binary metadata, which describes the format of the binary attachment, (that is, a text or binary data) the data of the specified type is copied into the payload field of the MQTT publish message.
  • :  Solace enterprise messaging APIs support the ability to carry structured data types (SDTs), such as maps and streams, in the binary attachment of the message or as user-defined message header fields. However, these SDTs cannot be used by MQTT clients. Therefore, they are not included in the MQTT publish message.

  • If the SMF message contains only an XML message payload, it will be copied into the payload field of the MQTT publish message.
  • If the SMF message contains both a binary attachment and an XML message payload, neither is sent—regardless of their content.
  • Custom user properties and userdata properties are not copied to MQTT publish message.

:  If the original SMF message contained a payload, but the process described above results in an MQTT publish message with no payload, the MQTT publish message is still delivered to the MQTT client even though it contains no payload. In this case, the message is noted as unformattable in the MQTT Session statistics.

REST Ingress, MQTT Egress

When an event broker receives a REST message, its payload, which consists of the data transmitted after the message’s HTTP headers, is delivered in its entirety in an MQTT publish message to an MQTT client. The particular Content-Type of the published message is not significant.

Will Messages

When connecting to an event broker, MQTT clients may specify a “last will and testament” (or simply “will”) message. A will message is stored with the MQTT session information that is allocated for the client, and it will be sent if an MQTT client is disconnected from the event broker unexpectedly.

A will message consists of a topic, QoS level, and message body. Will messages allow other interested MQTT clients about unexpected connection loss.

:  An event broker will not broadcast will messages when an MQTT session is terminated due to an event broker restart, high availability (HA) event broker failover, Message VPN shutdown, or Guaranteed messaging shutdown.

Retained Messages

Retained messages are used when publishers send state messages on an irregular and/or infrequent basis. Published messages which have their retain bit set are stored by the Solace messaging network and delivered to subscribing clients when they subscribe to a topic that matches the retained message’s published topic. This allows subscribers to be notified immediately of the most recent state upon subscribing.

For more information about configuring Solace PubSub+ event brokers to support retained messages, see Managing MQTT Retained Messages.

MQTT Retain Caches

In Solace PubSub+ event brokers, retained messages are preserved in special Cache Clusters called MQTT retain caches. In other words, Solace PubSub+ Cache is the underlying mechanism used to store, manage, and select which retained messages to publish as clients add topic subscriptions.

MQTT Retain Cache Ancillary Objects

When you create an MQTT retain cache, the event broker creates the following ancillary PubSub+ Cache objects.

Object Type Name Notes
Distributed Cache #retain-cache-<router-name> Each event broker (or pair, in HA deployments) constitutes its own distributed cache. For more information about distributed caches, refer to Distributed Caches.
Cache Cluster #retain-cache-cluster Every cache cluster deployed as a retain cache shares the same name. For more information about cache clusters, refer to Cache Clusters.
Cache Instance (Primary) #retain-cache-instance-<router-name>-primary Every event broker with a retain cache runs a retain cache instance. The primary instance runs on the primary router in an HA pair. For more information about cache instances, refer to PubSub+ Cache Instances.
Cache Instance (Backup) #retain-cache-instance-<router-name>-backup The backup cache instance is like the primary cache instance, except it runs on the backup router if redundancy is enabled.

Shared Subscriptions

Solace PubSub+ event brokers support OASIS Standard MQTT v5.0 shared subscriptions. You can use shared subscriptions to load balance large volumes of client data across multiple instances of back end data center applications. Each instance of the data center application joins the shared subscription, and the volume of data matching the shared subscription is randomly distributed between the application instances.

Consider the following when implementing shared subscriptions:

  • A MQTT QoS 0 topic subscription with the format $share/{ShareName}/{filter} indicates a shared subscription. Each component of the shared subscription is described as follows:
    • $share is a literal string that marks the subscription as being shared.
    • {ShareName} is a character string that does not include any "/", "+" or "#" characters.
    • {filter} is the remainder of the string which has the same syntax and semantics as a non-shared subscription.
  • Only QoS 0 shared subscriptions are supported. If a shared subscription with a higher QoS is requested, it is downgraded to QoS 0.
  • Shared subscriptions are supported only on non-durable sessions. A request to add a shared subscription to a durable MQTT session is rejected by the event broker.
  • In the OASIS Standard MQTT v5.0, a CONNACK property indicates whether an event broker supports shared subscriptions. Because CONNACK properties are not present in MQTT v3.1.1, the event broker cannot transmit this information to clients. This means that clients are required to know whether the event broker supports shared subscriptions and understand that shared subscriptions are treated as literal strings on brokers that do not support them.
  • Because allowing indiscriminate access to shared subscriptions is a potential security issue, you can specify which clients are able to use shared subscriptions. For more information, see Allowing Shared Subscriptions.
  • When using an ACL to control what clients can subscribe to, only the {filter} portion of a shared subscription is validated.
  • In MNR or DMR deployments with subscription exporting enabled, shared subscriptions are exported by default. In these cases, the event broker treats each exported subscription as a separate entity. In other words, clients connected to different nodes are not load balanced as a single group. To prevent exportation, a shared subscription must start with $noexport. For example, $noexport/$share/{ShareName}/{filter}.

Video: Shared Subscriptions Using MQTT

In this video, Leah Robert talks about Shared Subscriptions and demonstrates its application using MQTT subscription.