Viewing Guaranteed Messaging Information

You can use several show commands to view Guaranteed Messaging configurations and status on Solace PubSub+ event brokers.

Queue Information

To view the message queue configuration and status on the event broker for the message spooling mechanism used to support the persistent store notion of JMS queues and topics, enter the following User EXEC command.

solace> show queue <name> [message-vpn <vpn-name>] [flows | stats [priorities] | messages [oldest | newest] [msg-id <msg-id>] | replication-group-msg-id <replication-group-msg-id> | priority <priority>] | subscriptions | rates | sort-by-messages-spooled | sort-by-unacked-messages-spooled] | dead-message-queue <dmq-filter> [dmq-list] | replay] [durable | non-durable] [detail] [replay-state {initializing | active | pending-complete | failed | all}][flow <flow-id>] [count <num-elements>]

Where:

<name> is the full name of the message queue, or part of the queue name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all message queues.

<vpn-name> is the name of the Message VPN the queue belongs to.

flows specifies to show flows associated with the message queue spool.

stats specifies to show statistics associated with the message queue spool.

priorities specifies to show priorities along with the statistics associated with the message queue spool.

messages specifies to show information on the spooled messages associated with the message queue.

oldest specifies to show spooled messages in order from oldest to newest.

newest specifies to show spooled messages in order from newest to oldest.

msg-id specifies to show the spooled messages starting from the message identifier number specified by <msg-id>.

replication-group-msg-id specifies to show the spooled messages starting from the replication group message ID specified by <replication-group-msg-id>.

priority specifies to show only the messages in the message queue spool with the priority specified by <priority>. Valid values are 0 through 9.

subscriptions specifies to show the topic subscriptions added to the associated message queue.

rates specifies to show ingress and egress message rates associated with the queue.

durable specifies to show information on the active durable message queues.

non-durable specifies to show information on the active non-durable message queues.

detail specifies to show detailed information on the spooled messages.

flow <flow-id> specifies to show the message flow identifier associated with a given ingress or egress message flow. <flow-id> is the integer identifying the ingress or egress message flow to be displayed. Valid values are 0 through 16383.

sort-by-messages-spooled specifies to sort the display output in descending order for number of messages spooled for the given queue.

sort-by-unacked-messages-spooled specifies to sort the display output in descending order for number of unacked messages spooled for the given queue.

dead-message-queue <dmq-filter> specifies to show only those queues that use the given <dmq-filter> as their dead-message queue.

dmq-list specifies to list the queues associated with the dead message queue. Note that if the detail option is used, the dmq-list is ignored.

replay specifies to show replay log information.

durable specifies to show durable queues only.

non-durable specifies to show non-durable queues only.

detail specifies to show detailed information.

flow <flow-id> specifies to show queue details for the given Guaranteed Message FlowID. Valid values are 1 through 4294967295.

replay-state specifies to show only queues with an initializing, active, pending-complete or failed replay state.

count <num-elements> specifies the number of display elements to show. Valid values are 1 through 4294967295.

To display a detailed configuration and status summary screen for a message queue, enter the following command:

solace> show queue <name> message-vpn default detail

  • The status item ‘Deliver From’ under the Egress Flows section displays the Guaranteed message egress flow source as being delivered from either ‘input stream,’ ‘RAM,’ or ‘ADB or disk’.
  • When Guaranteed messages are being delivered immediately after being spooled, the egress flow is in the delivered from ‘input stream’ state. When Guaranteed messages are being delivered once the client is ready, thus requiring further processing, the egress flow is in either the delivered from ‘RAM’ or ‘ADB or disk’.
  • Guaranteed messages delivered from RAM require further processing, but are not necessarily copied to/from the ADB or disk (for example, queues with selectors, or queues with transient closures on their transport window).
  • Guaranteed messages delivered from input stream have lower latency than messages requiring further processing, while Guaranteed messages delivered from RAM have lower latency than messages delivered from ADB or disk.

To display a statistics summary screen for a message queue, enter the following command:

solace> show queue <name> stats

  • The item ‘Flow Id (State)’ under the Egress Flow Statistics section displays the Guaranteed message egress flow source as ‘Deliver from input stream,’ ‘Deliver from RAM,’ or ‘Deliver from ADB or disk’.
  • When Guaranteed messages are being delivered immediately after being spooled, the egress flow is in the ‘Delivered from input stream' state. When Guaranteed messages are being delivered once the client is ready, thus requiring further processing, the egress flow is in either the ‘Deliver from RAM’ or ‘Deliver from ADB or disk’ state.
  • Guaranteed messages delivered from RAM require further processing, but are not necessarily copied to/from the ADB or disk (for example, queues with selectors, or queues with transient closures on their transport window).
  • Guaranteed messages delivered from input stream have lower latency than messages requiring further processing, while Guaranteed messages delivered from RAM have lower latency than messages delivered from ADB or disk.

Clearing Queue Statistics

To clear the statistics for message queues on event brokers, enter the following Privileged EXEC command:

solace> enable
solace# clear queue <name> message-vpn <vpn-name> stats

Where:

<name> is the message queue name

<vpn-name> is the name of the Message VPN that the queue belongs to

Topic Endpoint Information

To view the topic endpoint configuration and status on the event broker for the message spooling mechanism used to support the persistent store notion of JMS queues and topics, enter the following User EXEC command.

solace> show topic-endpoint <name> [message-vpn <vpn-name>] [flows | stats [priorities] | messages [oldest | newest] [msg-id <msg-id>] | replication-group-msg-id <replication-group-msg-id> | priority <priority>] | topics | rates | sort-by-messages-spooled | sort-by-unacked-messages-spooled | dead-message-queue <dmq-filter> [dmq-list] | replay] [durable | non-durable] [replay-state {initializing | active | pending-complete | failed | all}] [detail] [flow <flow-id>] [count <num-elements>]

Where:

<name> is the full name of the durable topic endpoint, or part of the durable topic endpoint name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays durable topic endpoints.

<vpn-name> is the name of the Message VPN the topic endpoint belongs to.

flows specifies to show flows associated with the durable topic endpoint.

stats specifies to show statistics associated with the durable topic endpoint.

priorities specifies to show priorities along with the statistics associated with the durable topic endpoint.

messages specifies to show the spooled messages associated with the durable topic endpoint.

oldest specifies to show spooled messages in order from oldest to newest.

newest specifies to show spooled messages in order from newest to oldest.

msg-id <msg-id> specifies to show the spooled messages starting from the message identifier specified by <msg-id>.

replication-group-msg-id specifies to show the spooled messages starting from the replication group message ID specified by <replication-group-msg-id>.

priority specifies to show only spooled messages with the priority specified by <priority>. Valid values are 0 through 9.

topics specifies to show the topics associated with the durable topic endpoint.

rates specifies to show ingress and egress message rates associated with the topic endpoint.

sort-by-messages-spooled specifies to sort the display output in descending order for number of messages spooled for the given topic-endpoint.

sort-by-unacked-messages-spooled specifies to sort the display output in descending order for number of unacked messages spooled for the given topic endpoint.

dead-message-queue <dmq-filter> specifies to show only those durable topic endpoints that use the given <dmq-filter> as their dead-message queue.

replay specifies to show replay log information.

durable specifies to show information on the active durable message queues.

non-durable specifies to show information on the active non-durable message queues.

replay-state specifies to show only durable topic endpoints with an initializing, active, pending-complete or failed replay state.

detail specifies to show detailed information on the durable topic endpoint.

flow <flow-id> specifies to show the message flow identifier associated with a given ingress or egress message flow. <flow-id> is the integer identifying the ingress or egress message flow to be displayed. Valid values are 0 through 16383.

count <num-elements> specifies the number of display elements to show. <num-elements> is the integer specifying the number of elements. Valid values are 1 through 4294967295.

Clearing Topic Endpoint Statistics

To clear the statistics for topic endpoints on event brokers, enter the following Privileged EXEC command:

solace> enable
solace# clear topic-endpoint <name> message-vpn <vpn-name> stats

Where:

<name> is the name of the durable topic endpoint

<vpn-name> is the name of the Message VPN the durable topic endpoint belongs to

Message Spool Information

To view Guaranteed Messaging message status and usage, enter the following User EXEC command.

solace> show message-spool [message-vpn <vpn-name>] stats | detail | rates]

Where:

message-vpn <vpn-name> is the name of a Message VPN to show message spool information

stats specifies to show statistics on all spooled messages

detail specifies to show detailed information on the message spool facility

rates specifies to show information on ingress and egress subscription rates for active queues and topic endpoints

To display a standard status and usage summary screen for spooled messages, enter the following command:

solace> show message-spool

  • Always be aware of how fanout can impact the number of queue messages. For example, if one message is destined to be stored on 50 queues, then there are 50 queue messages to track. To determine what percentage of the queue message resources you are actually using, check the "Queue Message Resource Utilization" field of the show message-spool User EXEC command output.
  • The maximum amount of message spool disk space that can be used by the event broker (the Maximum Spool Usage) is a general measurement because there is also overhead stored for each message. To determine how much of the local disk storage capacity you are actually using, check the "Current Persistent Storage Usage (MB)" field of the show message-spoolUser EXEC command output.

To display a detailed status and usage summary screen for the message spool facility, enter the following command:

solace> show message-spool detail

The following table describes the detailed status and usage items for spooled messages displayed in the bottom portion of the summary screen.

Detailed Status and Usage Items for Spooled Messages

Item

Description

Values

Currently used and maximum message-spool entities section

Summarizes message spool entities being used to spool messages for clients, queues, and durable topic endpoints.

  • Integer

Total Flows section

Summarizes publisher (ingress) and subscriber (egress) Guaranteed message flows being used.

  • Integer

CVRID/LAG Config Status

Indicates whether the local CVRID/LAG configuration has been completed.

  • Unknown
  • Ready

ADB Status

Summarizes the high-level status of the flash memory card module and power module on the ADB, as well as the ADB itself.

The ADB status is by default ‘Not Ready’ if either one of its modules is ‘Not Ready”.

  • Not Ready
  • Ready

ADB Contents

Indicates whether the contents of the ADB are valid.

  • Invalid
  • Stale
  • Ready

Contents Key(s)

This is an internal signature used on the ADB to ensure synchronization sanity and prevent issues due to mis-configuration.

Only the ‘Local Contents Key’ is shown for non-redundant systems.

On Solace PubSub+ appliances:

  • IP address followed by two integers

On Solace PubSub+ software event brokers:

  • 32-bit number in dotted quad notation followed by two integers

Disk Status

Indicates the status of either the internal disk, or external disk storage array (as applicable).

  • Not Ready
  • Ready

Disk Contents

Indicates whether the spool-file directory on the disk is valid. Will generally be “Unknown” on a Solace PubSub+ event broker that is not active.

  • Unknown
  • Mount Error

    (This state is set if the WWN can not be mounted)

  • Invalid

    (This state is set if there is an ownership conflict with the disk contents.)

  • Ready

Disk Key(s)

This is an internal signature used on the ADB to ensure synchronization sanity and prevent issues due to mis-configuration.

Only the ‘Disk Key (Primary)’ key is shown for non-redundant systems.

On Solace PubSub+ appliances:

  • IP address followed by two integers

On Solace PubSub+ software event brokers:

  • 32-bit number in dotted quad notation followed by two integers

To display a detailed statistics summary screen for spooled messages, enter the following command:

solace> show message-spool stats

The following table describes the information logged in the detailed spooled message statistics.

Description of Detailed Spooled Message Statistics

Statistic

Description

Ingress Spool Discard Statistics:

Spooling Not Ready

Number of messages that could not be spooled and were discarded because the message spooling facility is not ready (for example, because the message spooling facility is not enabled or configured correctly through the CLI, or the ADB has detected a hardware fault).

Out of Order Messages

Number of messages that could not be spooled and were discarded because they were received out of order.

Duplicate Messages

Number of messages that could not be spooled and were discarded because they were duplicates.

No Eligible Destinations

Number of Guaranteed messages discarded because there is no existing endpoint that has a topic subscription matching the topic to which the message is published.

Spool Over Quota

Number of messages that could not be spooled and were discarded because the maximum allowed local message spool usage quota is exceeded.

Queue/Topic-Endpoint Over Quota

Number of messages that could not be spooled and were discarded because the maximum allowed message spool usage quota is exceeded.

Max. Message Usage Exceeded

Number of messages that could not be spooled and were discarded because the maximum allowed message usage count is exceeded.

Max. Message Size Exceeded

Number of messages that could not be spooled and were discarded because the maximum allowed message size is exceeded.

Remote Router Spooling Not
 Supported

Number of messages that could not be spooled and were discarded by remote event brokers because they do not have Guaranteed Messaging and message spooling activated through the message-spool Global CONFIG command.

Spool to ADB Fail

Number of messages that could not be spooled and were discarded because of a failure spooling to the ADB. For example, the Guaranteed messages could not be written to the ADB due to a hardware issue.

Spool to Disk Fail

Number of messages that could not be spooled and were discarded because spooling to the disk store failed (for example, due to being full)

Spool File Limit Exceeded

Number of messages that could not be spooled and were discarded because all spool file resources were used.

Errored Message

Number of messages that could not be spooled and were discarded due to detected errors.

Queue Not Found

Number of messages that could not be spooled and were discarded because the associated message queue could not be found (for example, because the message queue facility is not enabled or configured correctly through the CLI).

Spool Shutdown Discard

Number of messages that could not be spooled because ingress clients are disabled from publishing to the queue by the shutdownQueue CONFIG command.

User Profile Deny Guaranteed

Number of messages that could not be spooled because the user profile does not allow the publishing of Guaranteed messages.

Publisher Not Found

Number of messages that could not be spooled because the associated publisher could not be found.

No Local Delivery Discard

Number of messages discarded because they were sent to a queue or topic endpoint to which the same client as the message publisher is bound with No Local instruction.

TTL Exceeded

Number of messages discarded because the specified maximum Time to Live (TTL) value for a Message VPN bridge (as configured through the max-ttl Bridge CONFIG command) has been exceeded.

Publish ACL Denied

Number of messages discarded because the publish Access Control List (ACL) criteria has not been satisfied.

Destination Group Error

Number of Guaranteed messages discarded due to “reject-msg-to-sender-on-discard” being triggered on a different queue/topic endpoint in the same destination group. The destination group is the set of durable and/or temporary endpoints in the same Message VPN that have “reject-msg-to-sender-on-discard” turned on, as well as a topic subscription matching the topic to which the message is published.

Low-Priority-Msg Congestion Discard

Number of published low-priority messages that are discarded after the reject low‑priority message limit for an endpoint has been exceeded.

Replication Is Standby Discard

Number of messages discarded because Guaranteed publishing is not allowed when the Message VPN has a Replication standby state.

Sync Replication Ineligible Discard

Number of messages discarded because the event broker has been configured to reject messages when sync replication mode is ineligible.

Other

Reserved

Egress Spool Discard Statistics:

Messages Deleted

Number of spooled messages the ADB administratively deleted.

Message Expired To Discard

Number of spooled messages that expired and were discarded because no DMQ is configured.

Messages Expired to
#DEAD_MSG_QUEUE

Number of spooled messages that expired and were successfully moved to the DMQ.

Messages Expired to
#DEAD_MSG_QUEUE Failed

Number of spooled messages that expired but were not successfully moved to the DMQ.

These messages are discarded.

Max Redelivery Exceeded to
Discard

The number of spooled messages that were discarded after the maximum number of redelivered attempts were made.

Max Redelivery Exceeded to
#DEAD_MSG_QUEUE

The number of spooled messages that were moved to a DMQ after the maximum number of redelivered attempts were made.

Max Redelivery Exceeded to
#DEAD_MSG_QUEUE Failed

The number of spooled messages that were not able to be moved to a DMQ after the maximum number of redelivered attempts were made.

These messages are discarded.

TTL Exceeded to Discard

The number of messages discarded because the specified maximum Time to Live (TTL) value for a Message VPN bridge (as configured through the max-ttl Bridge CONFIG command) has been exceeded.

Message Processing Statistics:

Number of Ingress Messages

Number of spooled messages the ADB received after subscription matching has completed successfully.

     Promoted to Non-Persistent

Number of Direct messages changed to non‑persistent.

     Demoted to Direct

Number of persistent messages changed to Direct.

     Replicate Promoted

Number of Direct messages changed to non‑persistent and then sent to a Replicated Message VPN.

     Async Replicated

Number of messages in replicate async mode.

     Sync Replicated

Number of messages in replicate sync mode.

     From Replication Mate

Number of messages received from replicate mate.

Sequenced Topic Matches

Number of messages that matched a configured sequenced topic.

Sequence Number Already Assigned

Messages received with a sequence number already present, or that has matched multiple sequenced topics.

Sequence Number Rollover

Messages where the sequence number rolled over from (263 – 1) back to 1.

Sequence Numbered Messages
Discarded

Messages that matched a sequenced topic but were subsequently discarded.

Transacted Messages Not
Sequenced

Messages that matched a sequenced topic but did not have a sequence number applied because they were part of a transacted session.

Ingress Messages Discarded

Number of messages that could not be spooled and were discarded by the ADB at ingress.

Messages Spooled to ADB

Number of spooled messages sent to the ADB.

Messages Spooled To Disk

Number of spooled messages sent to disk for spooling.

Messages Retrieved From ADB

Number of spooled messages that were retrieved from the ADB. Deleted messages are also included in the count.

Messages Retrieved From Disk

Number of spooled messages retrieved from disk. Deleted messages are also included in this count.

Guaranteed Message Cache Miss

Number of messages that could not use optional NAB caching feature.

Messages Ingress Selector
Examined

Number of spooled messages examined by ingress selectors on topic endpoints.

Messages Selector Matched

Number of spooled messages that matched a selector.

Messages Selector Did Not Match

Number of spooled messages that did not match a selector.

Messages Egress Selector
Examined

Number of spooled messages examined by egress selectors on browser and consumer flows.

     Messages Selector Matched

Number of spooled messages that matched a selector.

     Messages Selector Did Not
     Match

Number of messages that did not match a selector.

Egress Messages Discarded

Number of messages discarded at egress.

Number of Egress Messages

Number of messages sent from the spool. Note that one message sent to multiple clients is counted as a single egress message.

     Redelivered

Number of attempts made to deliver a message to applications after the first delivery attempt from queues and topic endpoints. These additional attempts can result in the application receiving the same message more than once.

     Transport Retransmitted

The number of messages that were retransmitted by queues and topic endpoints at the transport layer (meaning from broker to Solace API) as part of a single delivery attempt. These retransmitted messages will not result in the application receiving the same message more than once, however, they may indicate a network issue.

Messages Confirmed Delivered

Number of spooled messages acknowledged by clients and neighboring event brokers.

     Store and Forward

Number of spooled messages that used store and forward delivery (the default) that are acknowledged by clients and neighboring event brokers.

     From Replication Mate

Number of spooled messages received from a replication mate that are acknowledged by clients and neighboring event brokers.

Transacted Session Processing Statistics:

Open Session

Number of attempts to open transacted sessions.

     Success

Number of transacted sessions successfully created.

     Max Sessions Exceeded

Number of times transacted sessions were not opened because the system limit for the maximum number of transacted sessions was exceeded.

     Other

 

     Success

Number of transacted sessions successfully created.

Transactions

Number of attempts to create local transactions.

     Success

Number of transactions successfully created. The number of transactions committed and rolled back are listed.

     Fail

Number of failures to create a transaction.

     Msgs Published

Number of messages published through a successfully committed transaction.

     Msgs Consumed

Number of messages consumed through a successfully committed transaction.

     Max Transactions Exceeded

Number of times the system limit for the maximum number of transactions was exceeded.

     Max Transaction Resources Exceeded

Number of times the system resources allocated for transactions was exceeded.

XA Transaction Processing Statistics:

Open Session

Number of attempts to open XA Sessions.

     Success

Number of transacted sessions successfully created.

     Max Sessions Exceeded

Number of times XA sessions were not opened because the system limit for the maximum number of XA sessions was exceeded

     Other

 

Transaction Requests

Number of XA requests made. Within this category, the number of requests that succeed and fail are provided. Each type of request that succeeds fails is also provided.

Msgs Spooled to ADB

Number of messages in XA transactions that were spooled to the ADB.

Msgs Retrieved from ADB or Disk

Number of messages in XA transactions that retrieved from the ADB or disk.

Msgs Published

Number of messages published through a successfully committed XA transaction.

Msgs Consumed

Number of messages consumed through successfully committed XA transactions.

Max Transactions Exceeded

Number of times the system limit for the maximum number of XA transactions was exceeded.

Max Transaction Resources      Exceeded

Number of times the system resources allocated for XA transactions was exceeded.

Clearing Message Spool Statistics

To clear the statistics for spooled messages on event brokers, enter the following Privileged EXEC command:

solace# clear message-spool stats

Client Message Spool Information

To show the client message spool configuration and status, enter the following User EXEC command:

solace> show client <name> message-spool [{ingress | egress} [flow <flow-id>]]

Where:

<name> is the full name of the client, or part of the client name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all clients.

ingress specifies to show ingress message spool information and statistics.

egress specifies to show egress message spool information and statistics.

flow specifies to show the message flow identifier associated with an ingress or egress message flow.

<flow-id> is the integer identifying the ingress or egress message flow. Valid values are 0 through 16383.

Client Message Spool Statistics

To show the client message spool statistics, enter the following User EXEC command:

solace> show client <name> message-spool-stats [{ingress | egress} [flow <flow-id>]]

Where:

<name> is the full name of the client, or part of the client name with the wildcard character ? used to represent one character of the name, or the wildcard character * used to represent zero or more characters of the name, where entering only the wildcard character * for the name displays all clients.

ingress specifies to show ingress message spool information and statistics.

egress specifies to show egress message spool information and statistics.

flow specifies to show the message flow identifier associated with an ingress or egress message flow.

<flow-id> is the integer identifying the ingress or egress message flow. Valid values are 0 through 16383.

Clearing Client Message Spool Statistics

To clear the message spool statistics database for all clients on an event broker, run the following Privileged EXEC command:

solace> enable
solace# clear message-spool-stats

XA Transaction Statistics

To show XA transaction statistics, enter the following User EXEC command:

solace> show xa-transaction [xid <xid>] [message-vpn <vpn-name>] [state <transaction-state>] [detail | sort-by-last-state-change | sort-by-messages-spooled]

Where:

<xid> is the Xid that uniquely identifies an XA transaction and the global transaction it is associated with. The wildcards * or ? may be used.

<vpn-name> is the name of the Message VPN that the XA transaction is in. The Message VPN name can contain from 1 to 32 characters. The wildcards * or ? may be used.

<transaction-state> specifies to show XA transactions with a specific state. If not specified, all states are included. The wildcards * or ? may be used.

detail specifies to show detailed information on the XA transactions.

sort-by-last-state-change specifies to sort the listed XA transactions according to the last state change in descending order.

sort-by-messages-spooled specifies to sort the display output for number of messages spooled in descending order.

The table below lists the information that may be displayed in spooled message statistics. Some of this information is only displayed when detailed statistics are requested.

XA Transactions Statistics

Statistic

Description

XA Session

The name of associated transacted session. If there is no associated session, then this field is
N/A.

Session ID

The Id of the associated transacted session. If there is no associated session, then this field is
N/A.

Idle Timeout

The number of seconds before an idle transaction may be automatically rolled back and freed.

Last State Change

The elapsed time since the last state change.

This does not include a status change from in-progress to complete, but only the time since entering the state.

State

The following states are possible for an XA transaction:

ACTIVE—The transaction is associated to a transacted session.

MARKED_ROLLBACK—The transaction is active and marked as rollback-only.

IDLE—The transaction is not associated to any transacted session and will be rolled back if not prepared, committed, or rolled back within the idle timeout period.

PREPARING—The transaction is processing a prepare request.

PREPARED—The transaction is prepared and ready for a commit or roll back request.

COMMITTING—The transaction is processing a commit request.

COMMITTED—The transaction is heuristically completed and successfully committed.

ROLLING_BACK—The transaction is processing a rollback request.

ROLLED_BACK—The transaction is heuristically completed and successfully rolled back.