Multi-Node Routing

The Multi-Node Routing feature allows multiple Solace routers to be networked together so that Direct messages published from clients connected to one router can be delivered to clients connected to other routers throughout the network. This effectively distributes the message routing load across multiple Solace routers in the network, resulting in aggregate message forwarding rates across the network that exceed the capacity of any one router.

Note:   

  • The Multi-Node Routing feature is for use with Direct messages only. Multi‑Node Routing does not route Guaranteed messages between Solace routers.
  • Network-wide load balancing with DTO, when used in conjunction with Multi-Node Routing, is not supported. A single node will always give priority to locally-connected clients when load balancing. Only when there are no available local clients eligible for DTO delivery will the router pass the DTO request to downstream routers. In such scenarios, no attempt will be made to load balance amongst multiple neighboring Solace routers.

To enable multiple Solace routers to work together effectively across a network in such a distributed message-routing environment, the Solace Content Shortest Path First (CSPF) protocol is used to link neighbor routers and allow them to discover their network neighbors and from that determine the complete message‑routing network topology to which they belong. Solace routers can then determine which neighbor is the optimal node for the forwarding of messages addressed to specific destination routers. This topology discovery is continuous and dynamically updated as Solace routers within the network go on or offline.

Multi-node routing also uses the Subscription Management Routing Protocol (SMRP) to enable linked neighbor routers to propagate topic subscriptions inserted by clients from one Solace router throughout the message‑routing network. This allows messages that are published to one router to be routed to other routers that also have connected clients interested in those topics.

Multi-Node Routing Compared to Message VPN Bridging

Messages published on one router can also be delivered to another networked router through the use of Message VPN Bridging. However there are some fundamental operational differences between the multiple-node routing and Message VPN bridging. The table below lists some of the differences between the two features.

After considering the differences between multi-node routing and Message VPN bridging listed above, some general recommendation on when to use each feature are provided. Note that both feature can be used in conjunction in a messaging network when carefully deployed. For more information on Message VPN bridging, refer to Working with Message VPN Bridges.

Comparing Multi-Node Routing and VPN Bridging

 

Multi-Node Routing

Message VPN Bridging

Supports Direct Messaging

Yes

Yes

Supports Guaranteed Messaging

No

Yes

Links only between Message VPNs with the same names

Yes

No. Bridges can be used to link Message VPNs with the same or different names.

All subscriptions in a Message VPN are used for topic matching

Yes

No. Only those subscriptions manually added to a bridge or queue associated with a bridge are used.

Topic Subscriptions are static

No. Topic subscriptions are dynamically updated as clients add or remove subscriptions.

Yes. Administrators manually add subscriptions to bridges (Direct) or queues associated with bridges (Guaranteed).

Supports linking of Message VPNs of Different Names

No. Only Message VPNs with the same name on separate routers can be linked.

Yes. Message VPNs with different names can be linked.

Supports linking of Message VPNs on the same router

No.

Yes. Message VPNs on the same router can be linked if they have different names.

You should use multiple-node routing when:

  • Guaranteed Messaging is not required
  • Message VPNs span multiple routers, with the same Message VPN names on each router
  • there is no need to limit the advertised subscriptions in a Message VPN (that is, subscriptions can be advertised to all routers in the network)

Other multiple-node routing advantages:

  • very simple to configure—the only configuration required is identifying the neighbor links between Solace routers and identifying which Message VPNs should advertise their subscriptions into the network
  • uses a common name for the Message VPN across the network
  • dynamically discovers the routers that are part of the same Message VPN
  • dynamically reroutes around failed or offline routers or failed inter-router links
  • guarantees a loop-free forwarding topology at all times
  • automatically propagates subscriptions for the Message VPN between routers in the network
  • subscriptions are automatically withdrawn from the network if all clients who subscribed to the topic have disconnected from the message router

You should Use Message VPN bridging when:

  • Guaranteed Messaging is required
  • there is a need to connect two distinct Message VPNs
  • there is a need to carefully control which subscriptions get advertised between Message VPNs on the routers

Note:  When Message VPN bridging, you should take care with complex network messaging configurations to ensure a loop-free forwarding topology and that subscriptions on bridge links do not attract excessive traffic for which there are no interested clients on the local router.

Related Provisioning and Configuration Information

For information on how to configure and manage routing links and how to enable subscriptions to be exported over those links, refer to Managing Multi-Node Routing Links.

Linking Routers

A routing link defines the route for connectivity between Solace routers. When building a messaging network, administrators must explicitly configure the routing links between Solace router neighbors. Using these bi-directional links, those neighbors are then able to exchange network topology using the CSPF protocol and subscription information using SMRP. These configured neighbor links allow Direct messages inserted into the network by a publisher client to travel between neighbors on their way to the final subscriber clients.

Compression can optionally used on neighbor router data connections (only data connections are affected).

Note:  Both ends of a message routing link must be configured. That is, between two Solace routers A and B, there must be a neighbor configuration in A for B, and a neighbor configuration in B for A.

Multiple bi-directional routing links can be used to route messages between routers in your network for load balancing or redundancy. The routing protocols used by the Solace Messaging Platform performs a shortest path first (SPF) calculation to determine the shortest path (as defined by link costs). Routes with lower link costs are preferred over those with higher link costs. If there are multiple routing links with equal cost paths, the a route with less hops will be chosen as the more preferred route.

Where alternative routes are possible, an administrator can assign explicit link costs to indicate the best inter-neighbor message route. The rationale for assigning a higher link cost to a particular route could be based on considerations such as the speed, latency, or monetary cost of the underlying communication link.

NOTICE: Change routing link costs only where alternative routes are possible over the existing routing links and you want to prefer one route over another route using the costs of individual links. Once the number of neighbor links required for your message routing network is determined, an administrator can decide where different costs for the routing links may be needed. For an example, refer to Routing Link Cost Examples.

Routing Link Cost Examples

If you want to prefer one route over another (for example, to specify a default link), the network administrator can change the routing link costs to set the preferred message route.

In the example shown inRouting Link Cost—Example 1, there are four bi-directional routing links available between Solace routers A, B, C, and D:

  • A-B routing link
  • B-C routing link
  • B-D routing link
  • C-D routing link

If the cost associated with the A-B routing link is set to 150, and the cost associated with the three other routing links (that is, B-C, B-D, and C-D) is set to 100, messages routed from Solace router A to C travel from Solace router A to Solace router B and then on to Solace router C.

Routing Link Cost—Example 1
Routing Link Cost-Example 1

However, as shown inRouting Link Cost Example 2 , if the cost associated with the A-B routing link is set to 150, while the cost associated with the B-C routing link is set to 250, and the cost associated with the C-D and B-D routing links is set to 100, messages routed from Solace router A to C travel from Solace router A to Site B, on to Solace router D, and then to Solace router C. Such a cost configuration may be desirable in cases where, for example, two Solace router sites in one city are connected by a high-speed link, while other overseas Solace router sites are connected by low-speed links.

Routing Link Cost Example 2
Routing Link Cost—Example 2

CSPF Neighbor Queues

Messages that are to be routed from a router to its neighbors over established routing links are passed through an internal neighbor TCP transmit queue (one for each neighbor). These are TCP transmit queues, which manage the flow of data that is transmitted from the router over the routing links to its neighbors.

Each of these neighbor queues have an associated maximum depth. The depths are measured in work units, whereby a work unit represents 2048 bytes of a message. The formula to convert a message size to number of work units is: NumWorkUnits = CEILING(message.length/2048

When a new message is being placed on the neighbor queue, its depth in work units is checked against the configured maximum depth of the queue. If the CSPF queue depth is at or above the configured maximum depth of the queue and new messages are received, the oldest messages currently queued may be discarded to make room for the newest received messages. When this happens a dataplane statistic for Egress Congestion Discards is incremented in this case.

Minimum Burst Length Tolerance

Note that a neighbor queue may temporarily exceed its maximum depth if it has a sufficiently‑sized minimum burst length tolerance. The minimum burst length tolerance ensures that a neighbor queue does not lose messages when it receives bursts of very large messages. It specifies the number of messages per queue that are always allowed entry into the queue, regardless of its current configured maximum depth.

For example, in a case where there are three messages currently on a queue, and the minimum burst length tolerance of the queue is set to 4, then one more message is allowed to be placed on the queue, regardless of whether the three current messages are large enough to fill the queue to above its maximum allowed depth. Thus, the configured minimum burst length tolerance of 4 allows the queue to temporarily exceed its configured maximum depth by allowing it to absorb a burst of large messages that would otherwise exceed the number of allowed work units.

Subscription Propagation and Management

When multiple-node routing is used, topic subscriptions and matching messages are only distributed between matching Message VPNs. This means that a router with a topic subscription in one message VPN can only attract matching messages from a neighbor router that were published to a Message VPN of the same name. For example if a message was published to Message VPN Blue on Router A, that message may be delivered to a client on Router B with a matching topic subscription in a Message VPN Blue, but it may not be delivered to a client on Router B with a matching topic subscription in a Message VPN Green.

Topic subscriptions can be added by external clients and internal clients (for example, the internal client of the Message VPN, #client). When external client subscriptions are associated with a specific virtual router name; internal client subscriptions are associated with a physical router name.

Exporting Topic Subscriptions

Each Message VPN has a topic subscription export policy associated with it. This export policy allows the administrator to determine whether or not topic subscriptions on the Message VPN get advertised to other Solace routers in the network. The default policy is to not export subscriptions, which means that clients connected to the router only receive messages on the Message VPN that were published by a client connected to the same router.

To receive messages from other Solace routers, the subscription export policy in a Message VPN must be set to export subscriptions. This causes all subscriptions added locally to the Message VPN to be advertised to the other routers in the network.