Managing Multi-Node Routing Links

This section discusses the tasks associated with defining and managing routing links between Solace routers so that topic subscriptions and Direct messages that clients add to one router can be delivered to clients connected to the same or other routers throughout the network.

To enable multiple-node routing, the Solace Messaging Platform relies on the Solace Content Shortest Path First (CSPF) protocol to enable routing links to be created between neighbor routers and distribute and update the message‑routing network topology. It also relies on the Subscription Management Routing Protocol (SMRP) to enable linked neighbor routers to propagate topic subscriptions inserted by clients from one Solace router throughout the message‑routing network.

Creating Neighbor Routing Links

  • To create a CSPF routing link to a neighbor Solace router, enter the following CONFIG command:

    solace(configure)# routing

    solace(configure/routing)# create cspf neighbor <physical-router-name> connect-via <connect-via> [control-port <control-port>] [compressed-data]

  • To edit the properties of an existing CSPF routing link to a neighbor Solace router, enter the following Routing CONFIG command:

    solace(configure)# routing

    solace(configure/routing)# cspf neighbor <physical-router-name>

Where:

<physical-router-name> is the name of the physical router. This name cannot begin with ‘V:’, as that indicates a virtual router.

<connect-via> is the static IP address and TCP port of the neighbor router in the form ‘IP address[:port]’, with the IP address specified in the dotted decimal notation form nnn.nnn.nnn.nnn, and the port specified as a decimal value from 0 to 65535 (for example, 192.168.100.1:55555). If port is unspecified, the default is 55555 for non-compressed data, and 55003 for compressed data.

<control-port> is the TCP control port number of the neighbor router. If left unspecified, the control port used is the one returned by the neighbor router during the neighbor link establishment phase. If specified as a decimal value from 0 to 65,535, then this value takes precedence over that returned by the neighbor router.

compressed-data specifies use compression across the neighbor router data connections. Only data connections are affected.

Entering the cspf neighbor Routing CONFIG command moves the CLI to a configuration mode for the CSPF link to the specified neighbor router. From here you can configure the cost for a routing link to the specified neighbor router.

Example:

solace(configure)# routing

solace(configure/routing)# create cspf neighbor solace2 connect-via 10.1.2.3

solace(configure/routing/cspf/neighbor)#

Note:   

  • The no version of this command (no cspf neighbor) deletes a CSPF protocol link on the given router to a neighbor router.
  • Both ends of a message routing link must be configured. That is, between two Solace routers A and B, there must be a neighbor configuration in A for B, and a neighbor configuration in B for A.

Configuring Link Costs

If there are multiple routes between neighbor routes, and you want to enforce one route over another route using link costs, you can assign specific costs to routing links, to reflect the differences in the speed or monetary cost of each underlying communication link. Links with lower costs are preferred over links with higher costs. For examples of how to calculate link costs, refer to Routing Link Cost Examples.

To change the cost for a routing link to a neighbor router, enter the following CONFIG command:

solace(configure/routing/cspf/neighbor)# link-cost <cost>

Where:

<cost> is the integer value representing the new link cost to the neighbor router. The valid range is 1 to 255. If this parameter is not provided, the default value is 100.

Note:   

  • The no version of this command (no link-cost) resets the link-cost value to the default.
  • Existing link costs can be viewed using the show cspf route User EXEC command.

Assigning Routing Interfaces

A routing interface must be assigned to use Solace routing protocols such as the CSPF and SMRP protocols that are used for multiple-node routing. A routing interface must also be assigned if the router is to be used in a high‑availability (HA) redundant pair or if Config-Sync is to be enabled.

To assign a routing interface, enter the following CONFIG command:

solace(configure)# routing

solace(configure/routing)# interface <phy-interface>

Where:

<phy-interface> is an ASCII string specifying the Ethernet interface port or LAG to be assigned. Valid values are eth<port> (for example, eth1); <cartridge>/<slot>/<port> (for example, 1/1/8); <cartridge>/<slot>/lag<N> (for example, 1/1/lag1). There is no default value.

Starting/Stopping Solace Routing Protocols

  • To start running Solace routing protocols on the Solace router, enter the following CONFIG command:

    solace(configure)# routing

    solace(configure/routing)# no shutdown

  • To stop Solace routing protocols from running on the Solace router (such as the CSPF protocol and SMRP used for the Multi-Node Routing feature), enter the following CONFIG command:

    solace(configure)# routing

    solace(configure/routing)# shutdown

Note:   

  • By default, Solace routing protocols are stopped (that is, they are not running on Solace routers).
  • To start the Multi-Node Routing feature, you must enter the no shutdown Routing CONFIG command on each Solace router being networked together.
  • Whenever a Solace router is restarted while running Multi-Node Routing, SMRP needs to learn of the topic subscriptions it is to become active for. It does this by synchronizing its database with the neighbor routers. However, if Solace routing protocols are stopped on any neighbor router, the router that is attempting to synchronize its SMRP database will stop its resynchronization attempt after three minutes.

Configuring TCP Settings for Routing Neighbors

For information on configuring TCP settings for neighbor routers, refer to Configuring TCP Settings. This section discusses configuring the TCP settings used for client‑to‑router and for router-to-router connections.

Viewing Multi-Node Routing Information

You can use show commands to view multi-node routing information.

Showing CSPF Neighbor Link Information

To show the configuration and state of CSPF neighbor links on the Solace router, enter the following User EXEC command:

solace> show cspf neighbor <physical-router-name> [stats [queues | detail] | connections [wide] | detail]

Where:

<physical-router-name> is the name of the physical neighbor router. This name cannot begin with ‘V:’, as that indicates a virtual router.

stats asks to show basic dataplane statistics on messages for the specified neighbor.

queues asks to show dataplane statistics on queued messages for the specified neighbor.

detail asks to show detailed dataplane statistics on messages for the specified neighbor.

connections displays TCP connection status information between the specified neighbor.

connections wide asks to show detailed TCP connection status information between neighbors in a wide-screen computer display format (300+ character width).

detail asks to show detailed information on the configuration and state of CSPF links.

Clearing CSPF Neighbor Link Stats

To clear CSPF message statistics related to neighbor links, enter the following commands:

solace> enable

solace# clear cspf neighbor <physical-router-name> stats

Where:

<physical-router-name> is the name of the physical neighbor router. This name cannot begin with ‘V:’, as that indicates a virtual router.

When this command is entered, all of the CSPF message statistics are reset to 0, and CSPF statistics begin to be recorded again from this point.

Showing SMRP Subscription Information

To view SMRP subscription information on the Solace router, enter the following User EXEC command:

solace> show smrp subscriptions [message-vpn <vpn-name>] [destination-name <destination-name>] [remote-router] [client] [queue] [topic-endpoint] [primary] [backup] [static] [{[dto-priority <priority>] [topic <topic-str>] [persistent] [non-persistent]} | {summary}]

Where:

message-vpn asks to display only the SMRP subscriptions to the specified Message VPN.

<vpn-name> is the full name of the Message VPN, or part of the Message VPN name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all Message VPNs.

destination-name asks to display only the SMRP subscriptions to the specified destination.

<destination-name> is the full name of the destination, or part of the destination name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all destinations.

remote-router asks to display only the SMRP subscriptions to remote routers.

client asks to display only the SMRP subscriptions to local clients.

queue asks to display only the SMRP subscriptions to local queues.

topic-endpoint asks to display only the SMRP subscriptions to local topic-endpoints.

primary asks to display only the SMRP subscriptions to primary local destinations.

backup asks to display only the SMRP subscriptions to backup local destinations

static asks to display only the SMRP subscriptions to static local destinations.

<priority> asks to display SMRP subscriptions for only the specified Deliver-To-One (DTO) client priority level. Valid values are P1, P2, P3, P4, or DA (for Delivery Always).

topic asks to display SMRP subscriptions for a topic subscription string

<topic-str> specifies the topic subscription string for which to display SMRP subscription information on in the form a/b/c

persistent asks to display only the SMRP subscriptions tor persistent topics.

non-persistent asks to display only the SMRP subscriptions to non-persistent topics.

summary asks to show the number of SMRP subscriptions for each destination.

Note:  In addition to Solace Message Format (SMF) topic subscriptions, the show smrp subscriptions User EXEC command may also display Message Queuing Telemetry Transport (MQTT) topic subscriptions. Because this command must be able to represent both SMF and MQTT topic syntax, if a topic subscription uses characters with special meaning for the topic syntax it uses (for example, wildcards), those characters may be displayed in the output as escaped characters.

Multi-Node Routing Link Configuration Example

NOTICE: Determining which pairs of Solace routers have neighbor link connections determines the topology of a network. However, network topology is a network engineering task, which is the responsibility of the customer. This example only describes the tasks required to configure Solace routers for a multiple-node routing link configuration once all the network engineering is done. If required, contact Solace for assistance with network engineering.

A multiple-node routing network consists of two or more Solace routers, with CPSF neighbor link connections provisioned between neighboring routers in accordance with the defined network topology.

To successfully provision a multiple-node routing network, two separate levels of configuration are required, in this order:

  1. Router-Wide Configuration Tasks
  2. Per-Neighbor Link Configuration Tasks

Note:  This section assumes that the basic router configuration tasks described in Managing Appliance Interfaces have been completed.

Router-Wide Configuration Tasks

Perform these tasks on each and every router in the network, regardless of the number of neighbor link connections, to provision and enable the routing interface used by the Solace routing protocols for multiple-node routing:

  1. To enable routers to connect to the routing interface of their neighbors, configure a static IP interface on each router in the multiple-node routing network. This static IP interface can either be an independent port on the NAB, or a LAG interface consisting of one or more NAB ports.

    solace> enable

     

    solace# configure

     

    solace(configure)# ip vrf msg-backbone

     

    solace(configure/ip/vrf)# create interface 1/6/1:3 static

    Create static IP interface

    solace(configure/ip/vrf/interface)# ip-address 192.168.1.10/24

    Set static IP interface address

    solace(configure/ip/vrf/interface)# no shutdown

    Enable the static IP interface

    solace(configure/ip/vrfinterface)# exit

     

    solace(configure/ip/vrf)# route default 192.168.1.1

    solace(configure/ip/vrf)# end

    solace#

    Configure the IP route for the Message Backbone VRF

    Note:  To assign independent IP addresses to all ports of the NAB (that is, have no LAG configured), or have a mixture of some NAB Ethernet ports grouped into a single LAG and the remaining ports independently addressed, refer to Managing Appliance Interfaces.

  2. Verify IP connectivity between the static IP addresses of neighboring routers by running the ping User EXEC command.

    For example, assuming the static IP interface address of a neighbor router is 192.168.1.11:

    solace(configure)# ping msg-backbone:192.168.1.11

    PING 192.168.1.11 56(84) bytes of data.

    64 bytes from 192.168.1.11: icmp_seq=1 ttl=20 time=0.105ms

    64 bytes from 192.168.1.11: icmp_seq=2 ttl=20 time=1.570ms

    64 bytes from 192.168.1.11: icmp_seq=3 ttl=20 time=0.098ms

    64 bytes from 192.168.1.11: icmp_seq=4 ttl=20 time=0.089ms

     

    --- 192.168.1.11 ping statistics ---

    4 packets transmitted, 4 received, 0% packet loss, time 4000ms

    rtt min/avg/max/mdev = 0.089/0.466/1.570/0.736 ms

  3. Assign and configure on each router the routing interface for use by the Solace routing protocols. This should be the same physical interface as the static IP interface already created in step 1, whether it be a single port on the NAB, or a LAG interface.

    Example

    solace> enable

    solace# configure

     

    solace(configure)# routing

    Enter routing configuration mode

    solace(configure/routing)# interface 1/6/1

    Configure the routing interface

    solace(configure/routing)# no shutdown

    solace(configure/routing)# end

    solace#

    Enable Solace routing protocols to run

Per-Neighbor Link Configuration Tasks

After successfully completing the preceding router-wide configuration tasks, do the following:

  1. Configure the CSPF neighbor link connections between neighbor routers.

    For example, assuming the neighboring routers are named solace10 and solace11, and they have static IP addresses of 192.168.1.10 and 192.168.1.11, respectively:

    solace10(configure)# routing

    solace10(configure/routing)# create cspf neighbor solace11 connect-via 192.168.1.11

     

    solace11(configure)# routing

    solace11(configure/routing)# create cspf neighbor solace10 connect-via 192.168.1.10

  2. Repeat the previous step for both sides of each and every CSPF neighbor link connection in the multiple-node routing network.

Configuring Neighbor Queues

CSPF neighbor queues are used to enqueue messages before they are sent over the data connection to neighbor routers. You can perform the following tasks for these queues:

Always contact Solace for technical support before you attempt to configure the CSPF neighbor queues on Solace routers. Failure to do so may result in data loss or service interruption due to unwanted secondary effects on router performance from use of the Routing Queue CONFIG level within the CLI.

Configuring Max Neighbor Queue Depths

If the CSPF neighbor queue depth is at or above the configured maximum depth of the queue and new messages are received, the oldest messages currently queued can be discarded to make room for the newest received messages. When this happens a dataplane statistic for Egress Congestion Discards is incremented in this case.

To modify the maximum depth of CSPF neighbor queues, enter the following CONFIG commands:

solace(configure)# routing

solace(configure/routing)# cspf queue

solace(configure/routing/cspf/queue)# max-depth <depth>

Where:

<depth> is the integer value representing the number of work units for the neighbor queues. (A work unit represents 2048 bytes of a message.) The valid range is 50 to 262144. The default value is 20000. Changing this value does not affect messages already successfully placed on the queue.

Note:  The no version of this command (no max-depth) resets the neighbor to the associated default.

Configuring Minimum Message Bursts

To configure the minimum number of messages that must be on a CSPF neighbor queue before the queue’s depth is checked against the maximum depth setting (thereby allowing the queue to absorb a burst of large messages that exceeds the number of allowed work units), enter the following CONFIG command:

solace(configure)# routing

solace(configure/routing)# cspf queue

solace(configure/routing/cspf/queue)# min-msg-burst <depth>

Where:

<depth> is the integer value representing the queue burst depth in messages. The valid range is 0 to 262144. The default value is 4. Changing this value does not affect messages already successfully placed on the queue.

The no version of this command (no min-msg-burst) resets the queue burst depth in messages to the default of 4.

Showing CSPF Queue Settings

To show the CSPF queue settings on the Solace router, enter the following User EXEC command:

solace> show cspf queue