Component Concept Maps

In this section we'll take you on a tour of the PubSub+ event broker's messaging components, and explain at a high level how they make data move from producers to the broker, and from the broker to consumers.

The PubSub+ broker provides a foundation for multi-protocol and standards eventing including Solace Message Format (SMF), JMS1.1, MQTT3.11, REST, and AMQP1.0. As events move through the broker, they are translated from an ingress messaging protocol to an egress messaging protocol for each consumer receiving messages. Given the range of protocol support, it's important to keep in mind that each open source protocol your app might make use of adds its own twist on how the broker's messaging components operate. First we'll talk about how Solace's own SMF protocol works, and then we'll discuss variations introduced by others.

On our tour we'll look at how things operate from the perspective of your application. So, let's get started with SMF.

SMF

We're going to show you how the PubSub+ SMF protocol makes use of the PubSub+ event broker's messaging components to move data from producers to the broker, and from the broker to consumers, all from the point-of-view of your app. There are a couple of ways we're going to do this:

  • There's a diagram, in the section SMF Component Diagram, where you can step your way from connection establishment to data flow, beginning from your app and moving through the broker's components,
  • And there's some supplemental information in SMF Component Operations where you can find a few more details about what's going on in those steps.

SMF Component Diagram

You can follow the process of connection establishment through to data flow in the diagram shown below:

  • Step through the process via SMF
  • Hover over the icon next to the circled numbers in the picture below to see information about actions at this location.

  • Find out more about the PubSub+ components that are involved
  • Clicking on icons associated with components will lead you to further information about that component.

Before you begin your walk-through, you should note that in the diagram the steps are categorized as follows:

  • Steps 1 through 5 deal with the operations the app and broker go through to establish a connection and get ready for event messaging.
  • Steps 6, 7, 9 and 10 focus on operations associated with sending and receiving event messages.
  • Step 8 deals with the establishment of application subscriptions.

 

1. Application initiates Solace API and Solace Context (Optional, can use default). 2. Application Create and config session, Username, password, broker DNS:Port, TLS, Compression: <br> a. session = JCSMPFactory.onlyInstance().createSession(properties) Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients. After a client is successfully authenticated, the ACL profile assigned to the client username used by the client, or the LDAP the authorization groups that the client belongs to (when LDAP authorization is used), is checked. 3. Application connect session:  <br> a. session.connect();  <br> b. Host initiates TCP session  <br> c. Session sends CONNECT message 4. Broker authenticates connection (happens automatically after step 3) 5. Broker authorizes connection (happens automatically after step 4): 
 a. Broker sends CONNECT ACK 6. Application SDK creates publisher flow:  <br> a. prod = session.getMessageProducer(..)  <br> b. SDK sends last messageID sent and last messageID ACK received  <br> c. Broker sends last MessageID receives and last messageID AQCK sent  <br> d. Application maintains this state as message flow to allow for correct re-transmissions as required on reconnects 7. When a message is published:  <br> a. prod.send(msg,topic)  <br> b. Application sdk sends message and decrements available window by one  <br> c. Event broker receives, routes and persists message  <br> d. Event broker sends ACK  1<br> e. Application SDK receives ACK and increments available window by one 8. (Optional, can be admin or direct subn) Application Subscribes by adding subscription to queue: <br> a. session.addSubscription(queue, topic, ..)  <br> b. Application sends SUBSCRIBE message with topic and endpoint 
 c. API sends message <br> d. Broker adds subscription to connection or backing queue based on subscription endpoint  <br> e. Broker can send SUBACK 9. Application creates consumer flow, endpoint name, windowing sizes: <br> a. cons = session.createFlow(listener, flow_prop) <b1<br> b. Application connect flow  <br> c. Application API handshakes flow and binds to Queue/Endpoint <br> d. Messages are then able to move from broker to subscriber client 10. Application starts flow: <br> a. cons.start() <br> b. Broker sends message and decrements available window by one <br> c. Client API receives message and passes to application <br> d. Client SDK sends transport ACK <br> e. Broker receives ACK and increments available window by one <br> f. Application sends Application ACK when finished with message <br> g. Broker receives Application ACK and deletes message from queue A client is only authorized to connect to a Message VPN that is associated with a client username that that client has been assigned. Message Virtual Private Networks (VPNs) are managed objects on PubSub+ message brokers that allow for the segregation of topic space and clients. Basic operation of guaranteed messaging. An overview of Solace API fundamentals. A client application can publish Guaranteed messages, that is, messages that the client application assigns a Persistent or Non‑Persistent delivery mode to Queue or Topic destinations To receive Guaranteed messages, client must create a consumer Flow within a Session, and bind that flow to an endpoint on a message broker that messages are published or attracted to. Understand the fundamentals of publishing and subscribing to topics.

SMF Component Operations

In this section we're going to elaborate a bit on the steps shown in the above diagram.

As we previously noted, steps 1 through 5 deal with the operations the app and broker go through to establish a connection and get ready for event messaging. The following sections provide a little more information on those steps:

Steps 6, 7, 9 and 10 focus on operations associated with sending and receiving event messages, and you can find a little more background information about them here:

Instantiating Solace API and creating a Solace Session

Your client applications instantiate the Solace API. From this point they can create Solace Contexts (which create a worker thread for event processing), and, subsequently, a Solace Session to connect to the broker. You have the flexibility to control the threading model by deciding how many sessions to add to a context. You can have multiple sessions per context, but only one context per session. If you need more threads to handle a single session, you'll need to pass event processing into an application worker thread pool. Also, a single session can only connect to a single broker instance at a time.

The Solace Session connection opens a TCP flow-controlled, bi-directional, connection from your application to the broker. This bi-directional session is the foundation of all communications for producers and consumers to send and receive messages for all Message Exchange Patterns (MEPs).

TCP Session properties

Fundamental properties of the TCP session are controlled via API session properties. This includes properties such as TCP (and possibly application) buffer sizes, and connection and reconnection behaviors. The session properties dictate the type of connection, be it plain-text, compressed ,or encrypted. Finally, session properties can control application keep-alives from the client to the broker, which are used to detect connection failures.

TCP Connection

Now that we've discussed how an application connects, let's talk about the connection itself and its behavior.

As previously mentioned, every connection that sends or receives data is a stateful bi-directional TCP connection. On top of the TCP connection is a messaging layer session, which begins with a connect message, or a data message that contains credential headers that are used during authentication, which is discussed in Client Authentication. Based on destination port and connection header properties in the connection request, the broker will expect Session properties such as compression, encryption, and so on.

Broker authorizes connection

Once a session is connected to the required services and authenticated, we move on to the authorization stage. Based on the client-username in the connection credentials, two profiles are applied: the Client Profile and the ACL Profile.

  • The Client Profile sets the broker resources the client can access. This includes configuration options like the ability to send and receive persistent messages, ability to create queues, and set how many system resource buffers the client’s connection can consume. For more information, refer to Client Profile Configuration.
  • The ACL Profile defines which data events the client can produce and consume.

Event processing: A Little More Detail

Now that we have an authenticated and authorized client connection, we're ready to send and receive messages.

For Direct event messaging this is quite simple, the client API contains a publish method to publish directly over the messaging session to send events and a subscribe directly to the messaging session connection to receive events. There are no messaging layer acknowledgments, timers, or re-transmissions, there're only TCP layer acknowledgments and retransmissions. If the TCP connection fails, and the messaging session is lost, then the events are also lost.

For Persistent event messaging things are a bit different in that there is messaging layer windowing, timers, acknowledgments, and retransmissions. This implies there's another layer of statefulness associated with persistent event publish and consumption. Each consumer state machine encapsulates an endpoint flow in the client API, and a Queue / Endpoint binding on the broker. These flows contain an endpoint, window, available window, last message sent, and last message acknowledged. This information allows the client to reconnect across sessions, and continue publish / consumption where the last session left off. If the Client-Profile allows for publishing persistent messages, there's at least one producer flow to allow publishing clients to send persistent message. If the Client-Profile allows for consuming persistent messages, there's a consumer flow per endpoint.

MQTT

Based on our understanding of the SMF's operation, now let's look at how MQTT service makes use of the PubSub+ event broker's messaging components to move data from MQTT producer to the broker, and from the broker to MQTT consumers. First, we'll look at the Differences between MQTT and SMF, and then examine the MQTT Component Diagram.

Differences between MQTT and SMF

  • VPN
  • There is no concept of a message VPN in MQTT, so for each unique message VPN created a new listen port needs to be assigned for each variant of MQTT traffic, MQTT/TCP, MQTT/WS, MQTT/TLS, MQTT/WSS.

  • QoS
  • For QoS0 and QoS1 application publication, the system behaves as it does for SMF direct and persistent messaging. Where QoS0 is like direct and QoS0 is like QoS1. Qos2 published messages are processed as QoS1 messages by the broker. For QoS1 subscriptions, a unique, dynamically created queue is created for each MQTT session, and subscriptions are added to that queue. This allows the queue to attract events, and the MQTT consumer can retrieve these events once they connect. Also, there's no handshake between the API and broker to learn the last message, or ack received on re-connect, which can lead to additional re-transmissions.

MQTT Component Diagram

You can follow the process of connection establishment through to data flow in the diagram shown below:

  • Step through the process via MQTT
  • Hover over the icon next to the circled numbers in the picture below to see information about actions at this location.

  • Find out more about the PubSub+ components that are involved
  • Clicking on icons associated with components will lead you to further information about that component.

Before you begin your walk-through, you should note that in the diagram the steps are categorized as follows:

  • Steps 1 through 4 deal with the operations the app and broker go through to establish a connection and get ready for event messaging.
  • Steps 5 and 7 focus on operations associated with sending and receiving event messages.
  • Step 6 deals with the establishment of application subscriptions.
1. Application initiates MQTT SDK: a. New MqttClient(host, clientName); b. Application Create and config session, broker DNS:Port, TLS, Compression. 2. Application connect session, Username, password: a. mqttClient.connect(connOpts); b. Host initiates TCP session c. Broker receives messaging layer connect. 3. Broker authenticates connections. 4. Broker authorizes connection. 5. When a message is published: <br> a. mqttClient.publish(topic, message); <br> b. Application sends PUBLISH message. c. SDK store message in memory or to disk. <br> d. SDK sends message. 
 e. Broker receives message, routes and persists according to QoS. <br> f. Broker sends PUBACK. <br> g. SDK deletes message. 6. When Applications subscribe: <br> a. mqttClient.subscribe(topic, 1); <br> b. Application sends SUBSCRIBE message. 1<br> c. SDK sends message. 1<br> d. Broker adds subscription to connection or backing queue based on subscription QoS. 1<br> e. Broker sends SUBACK. 7. When Broker sends a message: a. Broker sends PUBLISH message and decrements available window. <br> b. Client SDK receives message and passes to application. <br> c. Application sends PUBACK when finished with message. <br> d. Broker receives PUBACK and deletes message from queue. Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients.Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients. After a client is successfully authenticated, the ACL profile assigned to the client username used by the client, or the LDAP the authorization groups that the client belongs to (when LDAP authorization is used), is checked. Understand the fundamentals of publishing and subscribing to topics. A client is only authorized to connect to a Message VPN that is associated with a client username that that client has been assigned. Message Virtual Private Networks (VPNs) are managed objects on PubSub+ message brokers that allow for the segregation of topic space and clients. Basic operation of guaranteed messaging Learn more about MQTT Client at the Eclipse Paho JAVA documentation.

AMQP

The walk-through discussed in this section is a little different than that shown in previous ones. Those have been based on the Java language. This one is based on Node.js. For AMQP Java, the JMS2.0 programing interface seems to be standard and would be the preferred option. First, we'll look at the Differences between Solace implementation of AMQP and SMF, and then examine the AMQP Component Diagram.

Differences between Solace implementation of AMQP and SMF

  • Durability
  • Currently PubSub+'s AMQP solution only processes persistent messages.

  • VPN
  • There's no concept of a Message VPN in AMQP, so for each unique message VPN created, a new listen ports needs to be assigned for AMQP/TCP, AMQP/TLS, AMQP/WS, AMQP/WSS.

AMQP Component Diagram

You can follow the process of connection establishment through to data flow in the diagram shown below:

  • Step through the process via AMQP
  • Hover over the icon next to the circled numbers in the picture below to see information about actions at this location.

  • Find out more about the PubSub+ components that are involved
  • Clicking on icons associated with components will lead you to further information about that component.

Before you begin your walk-through, you should note that in the diagram the steps are categorized as follows:

  • Steps 1 through 6 deal with the operations the app and broker go through to establish a connection and get ready for event messaging.
  • Steps 8 and 9 focus on operations associated with sending and receiving event messages.
  • Step 7 deals with the establishment of application subscriptions.
Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients.Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients. After a client is successfully authenticated, the ACL profile assigned to the client username used by the client, or the LDAP the authorization groups that the client belongs to (when LDAP authorization is used), is checked. Message Virtual Private Networks (VPNs) are managed objects on PubSub+ message brokers that allow for the segregation of topic space and clients. Understand the fundamentals of publishing and subscribing to topics. A client is only authorized to connect to a Message VPN that is associated with a client username that that client has been assigned. 1. Application creates an AMQ10 Client: <br> a. new AMQP.Client(AMQP.Policy.merge({defaultSubjects : false})); 2. Application connects the client: <br> a. amqpClient.connect(pubsub+broker). 4. Broker authenticates connection, (Automatic from 3). 5. Broker authorizes connection, (Automatic from 4): <br> a. Broker sends CONNECT ACK. Application creates Sender: <br> a. amqpClient.createSender(queueName). <br> b. SDK creates a senderSession. 7. When a message is published: <br> a. amqpSender.send(message). <br> b. data placed in AMQP frame assigned a transferId. <br> c. data sent then increment its next-outgoing-id, decrement its remote-incoming-window <br> d. Broker will ack frame with disposition frame.    8. Application create Receiver.  <br> a. amqpClient.createReceiver(queueName).  <br> b. SDK creates a receiverSession 9. Application starts callback for messages and errors: <br> a. amqpReceiver.on('message',...) <br> b. data placed in AMQP frame assigned a transferId. <br> c. data sent then increment its next-outgoing-id, decrement its remote-incoming-window. <br> d. SDK will ack frame with disposition frame. Basic operation of guaranteed messaging.

JMS

Now that we've had a look at SMF's operation, we're in a position to use that to understand the twist JMS has on the operation of the broker's components. First, we'll look at the Differences between JMS and SMF, and then examine the JMS Component Diagram.

One other thing we'll point out is that the differences between JMS over SMF documented in this section are also the differences between JMS over AMQP.

Differences between JMS and SMF

You should also note that these differences, as well as specifying the differences between JMS over SMF, they are also the differences between JMS over AMQP.

  • Persistent and Non-Persistent publisher flows
  • The JMS specification mandates synchronous and asynchronous data movement. All messages published as persistent are sent on a publisher flow with window of one and are synchronous; non-persistent messages can be sent within a configurable window on a second publisher flow.

    For more information, take a look at this discussions on Persistent and Non-Persistent.

  • JNDI
  • To use a broker as the JNDI server, the JMS API lazily opens a second session to do JNDI lookups. Although this session isn't used to send data, it could be required to resolve JNDI names of the endpoints, and so on to complete the data send and receive functions.

    Further Information on JNDI service can found in Solace JNDI Objects.

  • Topic Endpoints
  • Although SMF supports topic endpoints, its topic-to-queue mapping features are much more powerful and are the recommended approach. JMS doesn't have a programmatic interface for topic-to-queue mapping and therefore recommends topic endpoints.

    For more information, refer to Endpoints.

JMS Component Diagram

You can follow the process of connection establishment through to data flow in the diagram shown below:

  • Step through the process via JMS
  • Hover over the icon next to the circled numbers in the picture below to see information about actions at this location.

  • Find out more about the PubSub+ components that are involved
  • Clicking on icons associated with components will lead you to further information about that component.

Before you begin your walk-through, you should note that in the diagram the steps are categorized as follows:

  • Steps 1 through 5 deal with the operations the app and broker go through to establish a connection and get ready for event messaging.
  • Steps 6, 7, 8, 9,10, and 11 focus on operations associated with sending and receiving event messages.
  • Step 9 deals with the establishment of application subscriptions.

For AMQP, context and queue names are created programmatically, and an example how to create them is shown here.

1. Application initiates JMS SDK which starts a single Solace Context. <br> a. Application Creates initialContext , initialContext(env) with username, password, broker URL. 2. (Optional, can be programmatically created) Application does JNDI lookups to find context factory information, initialContext.lookup(CONNECTION_FACTORY_JNDI_NAME): <br> a. This provides information such as messaging credentials and default message QoS, TLS, Compression. 3. Application connect connection to broker, connectionFactory.createConnection(): <br> a. Host initiates TCP session. <br> b. Broker receives messaging layer connect. 4. Broker authenticates connection. 5. Broker authorizes connection. 6. Application does JNDI lookups to find endpoint names, 1<br> initialContext.lookup(QUEUE_JNDI_NAME). 7. Application SDK creates producer: 1<br> a. session.createProducer(..); <br> b. SDK sends last messageID sent and last messageID ACK received. 1<br> c. Broker sends last MessageID receives and last messageID AQCK sent. <br>  d. Application maintains this state as message flow to allow for correct re-transmissions as required on reconnects. 8. When producer publishes a message: <br> a. producer.send(topic, message, DeliveryMode, Prioroty, TTL) <br> b. Application sdk sends message and decrements available window by one. <br> c. Event broker receives, routes and persists message. <br> d. Event broker sends ACK. <br> e. Application SDK receives ACK and increments available window by one. 9. (Optional for Topic Endpoints only) Application adds subscriptions: <br> a. session.createConsumer(topic); <br> b. Application sends SUBSCRIBE message with topic and endpoint. <br> c. SDK sends message.  <br> d. Broker adds subscription to connection or topic endpoint <br> e. Broker can send SUBACK. 10. (Optional can poll for message) Application create listener: <br> a. consumer.setMessageListener(..) 11. Application starts connection, which starts all consumers: <br> a. connection.start() b. Broker sends message and decrements available window by one. <br> c. Client SDK receives message and passes to application. <br> d. Client SDK sends transport ACK. <br> e. Broker receives ACK and increments available window by one. <br> f. Application sends Application ACK when finished with message. <br> g. Broker receives Application ACK and deletes message from queue. Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients.Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients. After a client is successfully authenticated, the ACL profile assigned to the client username used by the client, or the LDAP the authorization groups that the client belongs to (when LDAP authorization is used), is checked. Understand the fundamentals of publishing and subscribing to topics. A client is only authorized to connect to a Message VPN that is associated with a client username that that client has been assigned. Message Virtual Private Networks (VPNs) are managed objects on PubSub+ message brokers that allow for the segregation of topic space and clients. Creating durable topic subscribers. Basic operation of guaranteed messaging A Connection object is a client's active connection to its JMS provider. It typically allocates provider resources outside the Java virtual machine (JVM). A Session object is a single-threaded context for producing and consuming messages. Message consumers can receive messages from destinations using either Synchronous or Asynchronous mode. A JMSContext is the main interface in the simplified JMS API introduced for JMS 2.0. You can create message consumers to receive messages from a queue or for a specific topic.

REST

In the previous sections, we looked at SMF's operation along with JMS and MQTT, all driven by messaging APIs. We'll now take a look at how REST's standards based HTTP exchange patterns work with the broker's components. First, we'll look at the Differences between REST and SMF, and then examine the REST Component Diagram.

Differences between REST and SMF

Since REST messaging isn't driven by a messaging API, but instead by standards-based HTTP exchange patterns, there's a more significant list of differences between the base SMF and REST. However, at its core, the broker treats REST messages exactly the same with respect to authentication, authorization, routing, and persistence.

  • Publish
  • All API publishes are HTTP POSTs with expected ACKs. The POSTs are not pipelined in that each must carry the required authentication header and are individually authenticated. This is different than a messaging orientated protocol where the connection is authenticated with a connect messages containing authentication credentials and then the messages are pipelined on the connection.

  • Subscription
  • There is no subscribe type method in HTTP. All subscriptions are done by administrative actions.

  • Event delivery to consuming applications
  • A REST delivery endpoint is used to manage the connection from the broker to the application and deliver events via standard web-hooks http.

REST Component Diagram

You can follow the process of connection establishment through to data flow in the diagram shown below:

  • Step through the process via REST
  • Hover over the icon next to the circled numbers in the picture below to see information about actions at this location.

  • Find out more about the PubSub+ components that are involved
  • Clicking on icons associated with components will lead you to further information about that component.

Before you begin your walk-through, you should note that in the diagram the steps are categorized as follows:

  • Steps 1 through 6 deal with the operations the app and broker go through to establish a connection and get ready for event messaging.
  • Steps 8 and 9 focus on operations associated with sending and receiving event messages.
  • Step 7 deals with the establishment of application subscriptions.
Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients.Client profiles are associated with client username accounts so that you can easily apply common configurations to groups of clients. After a client is successfully authenticated, the ACL profile assigned to the client username used by the client, or the LDAP the authorization groups that the client belongs to (when LDAP authorization is used), is checked. Message Virtual Private Networks (VPNs) are managed objects on PubSub+ message brokers that allow for the segregation of topic space and clients. Understand the fundamentals of publishing and subscribing to topics. A client is only authorized to connect to a Message VPN that is associated with a client username that that client has been assigned. 1. Application Creates REST message: <br> a. Add Authentication Header. <br> b. Add Content-Type Header. <br> c. Add Solace-Reply-Wait-Time-In-ms if reply is required. <br> d. Add Message Body 2. Application Sends REST Message: <br> a. URI includes Solace broker URI with destination topic or queue. 3. Broker authenticates (Automatic from 2.): <br> a. URI includes Solace broker URI with destination topic or queue. 4. Broker authorizes (Automatic from 2.). 5. Broker routes (Automatic from 2.): <br> a. Based on URI. 6. Broker persists: <br> a. Broker ACKS (200 OK) either empty payload or reply body. 7. Application Subscribes: <br> a. This is not done programmatically. Administrators bind RDP to Queue containing subscriptions 8. Broker opens connection(s) to application via web-hooks: <br> a. Broker opens connection to application. b. Broker send message with HTTP POST. 9. Application API ACK message: <br> a. Broker deletes message.