Using MQTT

The Message Queuing Telemetry Transport (MQTT) protocol is a lightweight, open protocol that can be used for Machine to Machine (M2M) and Internet of Things (IoT) use cases.

Solace PubSub+, as of version 7.1.1, supports OASIS Standard MQTT v3.1.1. This allows client applications to inter-operate with the Solace PubSub+ without relying on Solace-specific APIs or custom software development.

Like the Solace Message Format (SMF) protocol, MQTT is a topic-based client/server, publish/subscribe protocol that enables client applications to exchange messages without direct knowledge of how those messages will be delivered to other interested clients.

Clients can publish messages to defined topics, and they may use topic filters (that is, topic subscriptions) to receive messages that are published to those matching topics. All topics are UTF-8 strings, which consist of one or more topic levels that are separated by forward slash “/” characters. Separating topic levels using slashes creates a hierarchy of information for organizing topics.

The SMF and MQTT protocols use similar topic syntax, therefore SMF and MQTT messaging applications can be used together in the same messaging network. That is, when an intersecting topic hierarchy is used, MQTT clients can receive messages published by non-MQTT clients (for example, SMF or REST clients), and non-MQTT clients can receive messages published by MQTT clients. However, there are some differences in topic syntax and usage between SMF and MQTT that must be considered before deploying MQTT applications in a network alongside SMF applications. For more information, see Topic Support & Syntax.

MQTT clients connect to an MQTT server (in this case, a message broker), which is responsible for maintaining their subscription sets and directing published messages to clients that have matching topic filters. Topic filters function the same as topic subscriptions in the Solace PubSub+, so for consistency, they will be referred to as topic subscriptions throughout this document.

An MQTT client connection to a specific Message VPN on a message broker is made through a dedicated MQTT port configured for the Message VPN that they are connecting to. The MQTT client connection also requires an MQTT session on the server. An MQTT session is represented as a managed object in Solace PubSub+, and it contains the session state for the MQTT client (that is, its subscriptions and messages).

  • Solace's implementation of MQTT complies with OASIS Standard MQTT v3.1.1.1. An annotated version of the specification (MQTT Version 3.1.1) that highlights any deviations, limitations, or choices made in the “SHOULD” and “MAY” clauses of the protocol specification for the Solace implementation. It is strongly recommended that network architects and programmers review this document.
  • MQTT is not supported on the Solace PubSub+ 3230 or Solace PubSub+ appliances that use a Network Acceleration Blade (NAB) model NAB-0401EM.

MQTT Sessions

An MQTT session object is a virtual representation of an MQTT client connection that exists as a managed object on a message broker. An MQTT session holds the state of an MQTT client (that is, it is used to contain a client’s QoS 0 and QoS 1 subscription sets and any undelivered QoS 1 messages).

An MQTT session can be created:

  • Automatically when a client successfully connects to the message broker.
  • By an administrator using the Solace CLI, SEMP, or SolAdmin. Although the MQTT specification does not require administrator‑provisioned MQTT sessions to be supported, they are allowed, and they provide more flexibility for application development.

Note:  MQTT sessions should not be confused with Solace sessions (that is, non‑MQTT sessions).

MQTT Session Persistence

When a connecting client provides a CLEAN=1 flag for the MQTT session, the client’s MQTT session and any related information (subscription sets and undelivered messages) are not persisted after the client disconnects. That is, the flag ensures that the session “cleans up” after itself and no information is stored on the message broker after the client disconnects. This is true even if the session was administratively provisioned (through CLI or SEMP). If the client provides a CLEAN=0 flag, the MQTT session is persisted on the message broker, which means that the client’s client ID, topic subscriptions, QoS levels, and undelivered messages are all maintained (that is, they are not cleaned up). The client may then reconnect to the persisted session later.

An MQTT session can be deleted:

  • automatically when a client that created the MQTT session with a CLEAN=1 flag disconnects
  • when a client creates a new MQTT session with a CLEAN=1 flag and the same session identifier as the previous session
  • manually by administrators through the Solace CLI, SEMP, or SolAdmin

Quality of Service Levels

MQTT publish messages and topic subscriptions are assigned separate quality of service (QoS) levels, which determine the level of guarantee applied to message delivery. The MQTT QoS levels are:

  • QoS 0—At most once delivery. No response is required by the receiver (whether it is a message broker or a subscriber), and no retry attempts are made if the message is not delivered.
  • The Solace equivalent to QoS 0 is a message delivery mode of Direct.

  • QoS 1—At least once delivery. This level ensures that the message is delivered at least once. In a QoS 1 exchange, the receiver (whether it is a message broker or a subscriber) is required to send an acknowledgment of the message to indicate that it has been received.
  • The Solace equivalent to QoS 1 is a message delivery mode of Guaranteed (that is, non‑persistent or persistent). For more information, see Guaranteed Messages.

  • A message broker requires a Guaranteed messaging configuration to provide QoS 1 service. If a message broker is not configured for Guaranteed messaging, all QoS 1 and QoS 2 MQTT topic subscriptions will be downgraded to QoS 0. Additionally, QoS 1 and QoS 2 messages will be accepted by the message broker, but they will be delivered as QoS 0 messages. For more information on how to configure a message broker for Guaranteed Messaging, see Guaranteed Messaging.
  • QoS 2—Exactly once delivery. Solace converts published QoS 2 messages to QoS 1.

The following figure shows how different QoS levels may be applied to the same message. From a publisher to a message broker, an MQTT publish message uses the QoS level assigned by the message publisher. From the message broker to the message subscriber, an MQTT publish message uses the QoS level set by the matching topic subscription for the consuming client.

QoS Levels Applied During Message Delivery
QoS Levels Applied During Message Delivery

When there is a topic subscription match for an MQTT publish message, a consuming client will receive the message with the lowest possible QoS level for its subscription. For example, if a message is published with a QoS of 1, and it matches the client’s QoS 0 subscription, it will be delivered with a QoS of 0. However, because MQTT uses a one-to-many pub/sub messaging model, that message could also be delivered to a matching QoS 1 topic subscription with a QoS of 1.

Queues for QoS 1 Suscriptions

An MQTT session that has QoS 1 topic subscriptions must be associated with a durable queue to hold those subscriptions and any undelivered messages that are attracted by those QoS 1 subscriptions.

When an MQTT session is automatically created by a client, a queue is created when the first QoS 1 subscription is added. If the MQTT session was created with CLEAN=1 CONNECT, the queue is deleted along with the MQTT session when the client disconnects, but if the MQTT session was created with CLEAN=0 CONNECT, the queue will remain after the client disconnects, and will only be deleted by administrative action.

When an MQTT session is administratively created, a queue is not created automatically for the MQTT session. Queues for administratively-created MQTT sessions must be manually created and deleted.

The configuration parameters given to an MQTT queue depends on whether it was created automatically on a client‑created MQTT session, or if it was created by an administrative action:

  • client‑created MQTT sessions—the queue uses the same configuration values used for standard dynamic client‑created queues. (For information on the default values that are used, see Configuring Default Values for Client-Created Endpoints.)
  • Note  
    • Unlike standard dynamically-created endpoints, an MQTT client cannot pass in custom endpoint properties and provision flags.
    • If the copy‑from‑on-create command is used to specify a CLI‑provisioned queue or topic endpoint with custom values, those values will also be applied to MQTT queues for client‑created MQTT sessions.
  • administratively created MQTT sessions—the queue uses the default configuration values for an queues provisioned through the Solace CLI with the exception that it is enabled (no shutdown) when created. For information on the default values used for queues provisioned by an administrator through the Solace CLI, see Configuring Queues.

MQTT Topics

As a publish/subscribe messaging protocol, MQTT relies on a hierarchical topics. Clients can publish messages on specific topics, and clients can receive published messages that match their current topic subscriptions.

MQTT topics are UTF-8 strings, which consist of one or more topic levels that are separated by forward slash “/” characters. Separating topic levels using slashes creates a hierarchy of information for organizing topics.

Connected clients can publish MQTT messages to defined topics, and they may use topic filters (that is, topic subscriptions) to receive messages that are published to those matching topics.

Payload Handling When Message Types Change

This section discusses how the payloads of MQTT publish messages received by the message broker (that is, ingress messages) are handled when they are subsequently delivered as different message types (either as SMF or REST messages) to non‑MQTT clients that have matching topic subscriptions. It also discusses how the message payloads of received SMF or REST message are handled when they are subsequently delivered as MQTT publish messages to consuming MQTT clients with matching topic subscriptions.

MQTT Ingress, SMF Egress

When a message broker receives an MQTT publish message from a client, the received message’s payload, which is a sequence of bytes, is encapsulated in the resulting egress SMF message as a binary attachment.

MQTT Ingress, REST Egress

When a message broker receives an MQTT publish message from a client, the received message’s payload is delivered with a Content-Type of application/octet‑stream.

SMF Ingress, MQTT Egress

An SMF message may contain binary data and XML data payloads, and in some cases, user-defined and Solace-defined message header fields.

When a message broker receives an SMF publish message from a client for which there is a matching MQTT topic subscription, the payload of the message is processed before it is sent to the MQTT client as an MQTT publish message.

The SMF message’s XML message payload or binary attachment can be used for the payload for the MQTT publish message, but not both.

  • If the SMF message contains only a binary attachment, the following occurs:
    • If there is no binary metadata, then the binary attachment is copied into the payload field of the MQTT publish message.
    • If there is binary metadata, which describes the format of the binary attachment, (that is, a text or binary data) the data of the specified type is copied into the payload field of the MQTT publish message.
  • Note:  Solace enterprise messaging APIs support the ability to carry structured data types (SDTs), such as maps and streams, in the binary attachment of the message or as user-defined message header fields. However, these SDTs cannot be used by MQTT clients. Therefore, they are not included in the MQTT publish message.

  • If the SMF message contains only an XML message payload, it will be copied into the payload field of the MQTT publish message.
  • If the SMF message contains both a binary attachment and an XML message payload, neither is sent—regardless of their content.
  • Custom user properties and userdata properties are not copied to MQTT publish message.

Note:  If the original SMF message contained a payload, but the process described above results in an MQTT publish message with no payload, the MQTT publish message is still delivered to the MQTT client even though it contains no payload. In this case, the message is noted as unformattable in the MQTT Session statistics.

REST Ingress, MQTT Egress

When a message broker receives a REST message, its payload, which consists of the data transmitted after the message’s HTTP headers, is delivered in its entirety in an MQTT publish message to an MQTT client. The particular Content-Type of the published message is not significant.

Will Messages

When connecting to a message broker, MQTT clients may specify a “last will and testament” (or simply “will”) message. A will message is stored with the MQTT session information that is allocated for the client, and it will be sent if an MQTT client is disconnected from the message broker unexpectedly.

A will message consists of a topic, QoS level, and message body. Will messages allow other interested MQTT clients about unexpected connection loss.

Note:  A message broker will not broadcast will messages when an MQTT session is terminated due to a message broker restart, high availability (HA) message broker failover, Message VPN shutdown, or Guaranteed messaging shutdown.

Retained Messages

Retained messages are used when publishers send state messages on an irregular and/or infrequent basis. Published messages which have their retain bit set are stored by the Solace messaging network and delivered to subscribing clients when they subscribe to a topic that matches the retained message’s published topic. This allows subscribers to be notified immediately of the most recent state upon subscribing.

For more information about configuring Solace PubSub+ message brokers to support retained messages, see Managing MQTT Retained Messages.

MQTT Retain Caches

In Solace PubSub+ message brokers, retained messages are preserved in special Cache Clusters called MQTT retain caches. In other words, PubSub+ Cache is the underlying mechanism used to store, manage, and select which retained messages to publish as clients add topic subscriptions.

MQTT Retain Cache Ancillary Objects

When you create an MQTT retain cache, the message broker creates the following ancillary PubSub+ Cache objects.

Object Type Name Notes
Distributed Cache #retain-cache-<router-name> Each message broker (or pair, in HA deployments) constitutes its own distributed cache. For more information about distributed caches, refer to Distributed Caches.
Cache Cluster #retain-cache-cluster Every cache cluster deployed as a retain cache shares the same name. For more information about cache clusters, refer to Cache Clusters.
Cache Instance (Primary) #retain-cache-instance-<router-name>-primary Every message broker with a retain cache runs a retain cache instance. The primary instance runs on the primary router in an HA pair. For more information about cache instances, refer to PubSub+ Cache Instances.
Cache Instance (Backup) #retain-cache-instance-<router-name>-backup The backup cache instance is like the primary cache instance, except it runs on the backup router if redundancy is enabled.

Shared Subscriptions

Solace PubSub+ message brokers support OASIS Standard MQTT v5.0 shared subscriptions. You can use shared subscriptions to load balance large volumes of client data across multiple instances of back end data center applications. Each instance of the data center application joins the shared subscription, and the volume of data matching the shared subscription is randomly distributed between the application instances.

Consider the following when implementing shared subscriptions:

  • Only QoS 0 shared subscriptions are supported. If a shared subscription with a higher QoS is requested, it is downgraded to QoS 0.

  • Shared subscriptions are supported only on non-durable sessions. A request to add a shared subscription to a durable MQTT session is rejected by the message broker.
  • In the OASIS Standard MQTT v5.0, a CONNACK property indicates whether a message broker supports shared subscriptions. Because CONNACK properties are not present in MQTT v3.1.1, the message broker cannot transmit this information to clients. This means that clients are required to know whether the message broker supports shared subscriptions and understand that shared subscriptions are treated as literal strings on brokers that do not support them.
  • Because allowing indiscriminate access to shared subscriptions is a potential security issue, you can specify which clients are able to use shared subscriptions. For more information, see Allowing Shared Subscriptions.
  • When using an ACL to control what clients can subscribe to, only the {filter} portion of a shared subscription is validated.
  • In MNR or DMR deployments with subscription exporting enabled, shared subscriptions are exported by default. In these cases, the message broker treats each exported subscription as a separate entity. In other words, clients connected to different nodes are not load balanced as a single group. To prevent exportation, a shared subscription must start with $noexport. For example, $noexport/$share/{ShareName}/{filter}.

Video: Shared Subscriptions Using MQTT

In this video, Leah Robert talks about Shared Subscriptions and demonstrates its application using MQTT subscription.