Monitoring PubSub+ Cache Configuration and Operations

You can use the following show commands to monitor the PubSub+ Cache configuration and operational status:

Show Product Key
Show Distributed Cache
Show Cache Cluster
Show Cache Instance

Show Product Key

To view the system product keys and features that they unlock, enter the following User EXEC command:

solace> show product-key

Show Distributed Cache

To view the configuration and status of a Distributed Cache on the event broker, enter the following User EXEC command:

solace> show distributed-cache {<name> [message-vpn <vpn-name>] [detail] | summary}

Where:

<name> is the Distributed Cache name

<vpn-name> is the name of the associated Message VPN

detail asks to show detailed information for the Distributed Cache

summary asks to show a summary of all Distributed Caches, their Cache Clusters, and associated PubSub+ Cache Instances.

The cacheManagementUp attribute reflects the operational state of distributed cache management. The state will be up if the following conditions are satisfied:

DR/Replication for the Message VPN is either disabled or DR-active
The virtual router associated with the Distributed Cache is in an active role for the associated Message VPN
The Distributed Cache is enabled

Show Cache Cluster

To view the contents of a Cache Cluster under a Distributed Cache on the event broker, enter the following User EXEC command:

solace> show cache-cluster <name> [distributed-cache <cache-name>] [message-vpn <vpn-name>] [detail | topics [filter <topic-pattern>]]

Where:

<name> is the Cache Cluster name

<cache-name> is the name of the Distributed Cache the Cache Cluster belongs to

<vpn-name> is the name of the associated Message VPN the Cache Cluster belongs to

<cluster-name> is the name of the associated Cache Cluster

detail asks to show detailed information for the Cache Cluster

topics asks to show information on topics the Cache Cluster is responsible for

filter <topic-pattern> asks to filter the topic information based on the specified topic string pattern

Examples:

solace> show cache-cluster *

                                                  Admin  Instances  Config Lost
Cluster Name     Cache Name       Message VPN     Status (up/conf)  Topics Msg
---------------- ---------------- --------------- ------ ---------- ------ ----
eq-cluster1      equities         default         Up       2 / 2         1 No
eq-cluster2      equities         default         Up       0 / 1         1 No
eq-cluster3      equities         default         Up       0 / 1         1 No
fi-cluster1      fixed-income     default         Up       0 / 4         0 No
fi-cluster2      fixed-income     default         Up       0 / 0         0 No
fi-cluster3      fixed-income     default         Up       0 / 0         0 No

solace> show cache-cluster eq-cluster1 detail

Cache Cluster               : eq-cluster1
  Distributed Cache         : equities
  Message VPN               : default
  Admin Status              : Up
  Deliver-To-One Override   : Enabled
  Max Memory (MB)           : 2048
  Max Msgs Per Topic        : 1
  Max Topics                : 2000000
  Message Lifetime          : Unlimited
  New Topic Advertisement   : Disabled
  Request Queue Depth       : 100000
  Instances (up)            : 2
  Instances (configured)    : 2
  Topics (configured)       : 1
  Lost Message              : No
  Global Caching
    Heartbeat Interval (sec)    : 3
    Topic Lifetime (sec)        : 3600
    Admin Status                : Disabled
    Topic Prefixes (configured) : 3
    Home Cache Clusters         : 2
    abc
    xyz
 
Event Threshold              Set Value    Clear Value
---------------------------- ------------ ------------
Data Byte Rate (bytes/sec)   2000000000   1500000000
Data Message Rate (msgs/sec) 48000        36000
Max Memory (%)               80           60
Max Topics (%)               80           60
Request Queue Depth (%)      80           60
Request Rate (msgs/sec)      80000        1000
Response Rate (msgs/sec)     80000        1000

Show Cache Instance

To view the contents of local and remote PubSub+ Cache Instances under a Distributed Cache on the event broker, enter the following User EXEC command:

solace> show cache-instance <name> [cache-cluster <cluster-name>] [distributed-cache <cache-name>] [message-vpn <vpn-name>] [detail | remote {status | home-cache-clusters | topics [detail] [filter <topic-pattern>] [type {local | global}]}]

Where:

<name> is the PubSub+ Cache Instance name

<cluster-name> is the name of the Cache Cluster that the PubSub+ Cache Instance belongs to

<cache-name> is the name of the Distributed Cache that the PubSub+ Cache Instance belongs to

<vpn-name> is the name of the Message VPN that the PubSub+ Cache Instance belongs to

detail shows detailed information on the specified PubSub+ Cache Instance

remote shows the statistics of the PubSub+ Cache Instances managed by the Designated Router over the message bus

status shows status information on the PubSub+ Cache Instances

home-cache-clusters shows remote global caching home-cache-clusters status

topics shows information on topics cached by the PubSub+ Cache Instances

detail shows a more detailed display of each topic

filter <topic-pattern> filters the topic information based on the specified topic string pattern. Filter topics, can contain wildcards * or ?

type shows only topics of the specified type

local shows only local topics

global shows only global topics

For information on the possible administrative and operational states that a PubSub+ Cache Instance could be in, refer to PubSub+ Cache Instance States.

Examples:

solace> show cache-instance *

                                          Admin  Oper     Auto  Stop     Lost
Cache Instance>      Cache-Cluster        Status Status   Start On Loss  Msg
-------------------- -------------------- ------ -------- ----- -------- ---
fi-cluster1-instance fi-cluster1          Up     NotAvail Yes   Yes      No
fi-cluster1-instance fi-cluster1          Down   NotAvail No    No       No
fi-cluster1-instance fi-cluster1          Up     NotAvail Yes   Yes      No
eq-cluster1-instance eq-cluster1          Up     Up       Yes   Yes      No
eq-cluster1-instance eq-cluster1          Up     Up       Yes   Yes      No
eq-cluster2-instance eq-cluster2          Up     NotAvail Yes   Yes      No

solace> show cache-instance eq-cluster1-instance1 detail

Cache Instance          : eq-cluster1-instance1
  Cache Cluster         : eqcluster1
  Distributed Cache     : equities
  Message VPN           : default
  Admin Status          : Up
  Operational Status    : Up
  Auto Start            : Yes
  Stop On Lost Message  : Yes
  Last Stopped          : Never
  Reason For Last Stop  :
  Lost Message          : No
    Last Set Rxd        : Never
    Last Clear Txd      : Never
  Reg With Appliance    : Feb 19 2012 20:39:05
  Last Heartbeat Rxd    : Feb 20 2012 15:39:35

solace> show cache-instance instance1 remote status

Cache Instance                     : instance1
  Cache Cluster                    : cluster1
  Distributed Cache                : dist-cache-001
  Message VPN                      : default
  Operational Status               : Up
  Cache Software Version           : 5.5.0.1
  Cache Software Build Date        : May  4 2012 13:22:35
  API Version                      : 5.5.0.3
  API Platform                     : Linux26-x86_64_opt
  User                             : 'root'  Computer: 'perf-128-61'  Process ID: 1992
  Plugin Description               : Solace Default Plugin v1.0
  Plugin Status                    : DISABLED
  Connected To Host                : host '192.168.164.80', IP 192.168.164.80:55555
  Client                           : instance1
  Current Time                     : May 24 2012 13:58:30 EDT
  Process Start Time               : May 24 2012 11:23:15 EDT
  Last Connection Uptime           : May 24 2012 11:23:15 EDT
  Last Connection Downtime         : Never
  Total Mem Utilization (MB)       : 101
  CPU Utilization (%)              : 2
  Heartbeat Requests Txd           : 974
  Heartbeat Responses Rxd          : 974
  Global Caching
    Heartbeat Requests Txd         : 0
    Heartbeat Responses Rxd        : 0
    Home Clusters Connected        : 0
    Home Clusters Configured       : 0
    Suspect Topics                 : 0
  New Topic Advertisments Txd      : 0
  Config Updates Rxd               : 13
  Invalid Requests Rxd             : 0
  Cache Requests Rxd               : 3860
    Local Topics                   : 3860
    Global Topics                  : 0
    Global Topics Forwarded        : 0
    Timeout                        : 0
  Cache Responses Txd              : 3860
    OK                             : 3343
    No Match                       : 517
    Timeout                          : 0
  Lost Message                     : No
    Last Set                       : Never
    Last Clear Rxd                 : Never
  Last Clear Stats Rxd             : Never
  Last Backup Of Cached Messages
    Last Successful Backup         : Never
    Last Status                    : N/A
    Last Filename                  : N/A
  Last Restore Of Cached Messages
    Last Successful Restore        : Never
    Last Status                    : N/A
    Last Filename                  : N/A
 
  Current Value     Peak Value
  ------------- --------------
  Incoming Data Msg (1 sec sample) (msgs/s)              9993          10208
  Incoming Data Msg (60 sec sample) (msgs/s)             9999           9999
  Incoming Data Byte (1 sec sample) (bytes/s)          749475         765600
  Incoming Data Byte (60 sec sample) (bytes/s)         749978         749982
  Outgoing Data Msg (1 sec sample) (msgs/s)                10            500
  Outgoing Data Msg (60 sec sample) (msgs/s)                2             58
  Cache Request (1 sec sample) (msgs/s)                     1              1
  Cache Request (60 sec sample) (msgs/s)                    0              0
  Cache Request Queue Depth                                 0              1
  Cached Topics                                            10             50
    Local Cached Topics                                    10             50
    Global Cached Topics                                  n/a            n/a
  Messages Cached                                          10            611
    Local Cached Messages                                  10            611
    Global Cached Messages                                n/a            n/a

  Message Bytes Cached                                    750          45825
 
  Messages Cached (cumulative)                       77081139
  Message Bytes Cached (cumulative)                1486118129
  Message Lost Indications Rxd                              0
  Messages Discarded (since last lost clear)                0
    Parse Error (lost message)                              0
    Too Big Error (lost message)                            0
    Max Topics (lost message)                               0
    Max Memory (lost message)                               0
    Cluster Sync Fail (lost message)                        0
    Cluster Sync Suspect (lost message)                     0
    P2P Topic Filter                                        0

As shown by the above example, status requests display cache request and response time-out statistics.
When a messaging API sends a cache request, it indicates the amount of time it is willing to wait for a response. When the PubSub+ Cache Instance receives the cache request, it timestamps it and inserts it into the request queue. When the PubSub+ Cache Instance eventually extracts the cache request from the request queue, it checks to see how long the request has been in the queue. If this time exceeds the time that the API is willing to wait for a response, then the request is discarded and the Cache Requests Rxd time-out statistic is incremented.
When a cache request is too large to handle in one cache response, the PubSub+ Cache Instance responds with the “more” flag set. If the API does not send a ‘get-next cache-request’ within the allotted session time-out (the default is 10 seconds), the cache request session is canceled, and the Cache Responses Txd time-out statistic is incremented.

solace> show cache-instance eq-cluster1-instance1 remote topics

Cache Instance               : eq-cluster1-instance1
Cache Cluster              : eq-cluster1
Distributed Cache          : equities
Message VPN                : default
 
           Current   Cumulative Current    Cumulative Last
Topic             Msg Count Msg Count  Byte Count Byte Count Arrival
---------------- ---------- ---------- ---------- ---------- ---------------
md/nyse/bac              1       2000       1026    2052000  Feb 26 2012 06: 24:18
md/nyse/c                1       1000       1024    1024000  Feb 26 2012 06: 24:18
md/nyse/ge               1       1000       1025    1025000  Feb 26 2012 18: 22:12
md/nyse/pfe              1       1000       1026    1026000  Feb 26 2012 06: 24:1
---------------- ---------- ---------- ---------- ---------- ---------------
Total                    4       5000       4101    5127000

If you are using Global Caching, you can enter the following User EXEC command to display the status of the home Cache Clusters that a PubSub+ Cache Instance can request global topics from.

solace> show cache-instance v1.dc21.Local1.instance4 remote home-cache-clusters

Cache Instance                     : v1.dc21.Local1.instance4
  Cache Cluster                    : v1.dc21.Local1
  Distributed Cache                : v1.dc21
  Message VPN                      : v1
  Global Heartbeats Txd            : 21010
 
Home Cache Cluster                 : v1.dc11.Home1
  Heartbeat Status                 : Connected
  Heartbeats Received              : 20310
  Last Heartbeat Timeout           : Never
  Cache Requests Rxd               : 552901
  Cache Requests Forwarded         : 250900
    OK                             : 250900
    Fail                           : 0
    No Match                       : 0
    Timeout                        : 0

PubSub+ Cache Instance States

This section provides information on possible administrative and operational states that a PubSub+ Cache Instance could be in. These states can be viewed through many show cache-instanceUser EXEC commands (refer to Show Cache Instance).

PubSub+ Cache Instance Administrative State
PubSub+ Cache Instance Operational State
Lost Message State

PubSub+ Cache Instance Administrative State

The table below lists the administrative states that can appear for a PubSub+ Cache Instance that has established communications with the Designated Router. These states are displayed in the show cache-instance User EXEC command output.

PubSub+ Cache Instance Administrative States
Administrative State	Description
down	The PubSub+ Cache Instance is running, is connected to the message bus, and can communicate with the Cache Manager on the Designated Router. However, all of its topic subscriptions are removed from the network. Thus, a PubSub+ Cache Instance in the down state neither receives nor caches any messages, and it neither receives nor responds to cache requests. A PubSub+ Cache Instance can enter the down state when the shutdown command is issued for it or its parent Cache Cluster or Distributed Cache.
stopped	The PubSub+ Cache Instance is connected to the message bus, and it can receive and cache messages, but it cannot receive nor respond to cache requests. The typical cause for a stopped state is that the PubSub+ Cache Instance is not configured for auto-start, and after the PubSub+ Cache Instance is enabled, it enters this state. Thus, it is awaiting for an admin start command (refer to Performing PubSub+ Cache Administrative Tasks). A PubSub+ Cache Instance can also enter this state if cluster synchronization fails or if the PubSub+ Cache Instance is configured for ‘stop-on-lost-message’ and a ‘lost-message’ event occurs.

PubSub+ Cache Instance Operational State

The table below lists the potential operational states that can appear for a PubSub+ Cache Instance that has established communications with the Designated Router. These states are displayed in the show cache-instance User EXEC command output.

PubSub+ Cache Instance Operational States
Operational State	Description
registered	The PubSub+ Cache Instance has successfully registered and authenticated with the Designated Router. If a PubSub+ Cache Instance fails to register, an event log is generated.
config-sync	The PubSub+ Cache Instance is synchronizing its configuration with the Designated Router. A config-sync state can occur when the configuration is pending confirmation from the PubSub+ Cache Instance back to the Designated Router.
cluster-sync	The PubSub+ Cache Instance is fetching cached messages from or providing cached messages to another PubSub+ Cache Instance in the Cache Cluster.
up	The PubSub+ Cache Instance is caching messages that match configured topic subscriptions and is ready to respond to cache requests.
unknown	The PubSub+ Cache Instance has not been heard from.
invalid	A mismatched configuration exists between the Designated Router and one or more of the PubSub+ Cache Instances attempting to register with it.

Lost Message State

A PubSub+ Cache Instance can enter a Lost Message state, which indicates in the Cache Request Responses that the data integrity may be suspect. A PubSub+ Cache Instance can enter a Lost Message state under any of the following conditions:

The router indicates that messages were lost in transit through the message bus.
Local discards on ingress messages to PubSub+ Cache service occurred. This could occur, for example, due to exceeding memory or topic limitations, invalid format or size.
The PubSub+ Cache Instance disconnected from the message bus and subsequent reconnects occurred without a clean Cache Cluster synchronization.
A custom Ingress Message Plug-in failure occurred. For information on the Ingress Message Plug-in, refer to Using Ingress Message Plug-Ins.

Failure to add topic subscriptions to the message bus is not considered an indication of lost messages. In such a case, an event log is generated.

On entering a Lost Message state, an event message is generated to indicate back to the Designated Router that a message has been lost. And to avoid high volumes of lost message events while in this state, no further lost message events are sent back to the Designated Router. (One side effect of this is that subsequent discard notifications are masked.)

To clear the Lost Message state, an administrative clear-event action must be performed for the Distributed Cache. (Refer to Performing PubSub+ Cache Administrative Tasks.)

Administrative actions do not mark the PubSub+ Cache Instance as having lost messages. Therefore the following actions do not affect the Lost Message state:

Deletion of messages through the use of either a scheduled-delete-message Distributed Cache CONFIG command (refer to Scheduling Message Deletes) or an administrative delete-messages Distributed Cache Admin EXEC command (refer to Performing PubSub+ Cache Administrative Tasks).
Shutdown, then later re-enable the Distributed Cache or PubSub+ Cache Instances.
Topic subscription churn.
Startup of a new PubSub+ Cache Instance process. (The new PubSub+ Cache Instance still synchronizes with the existing PubSub+ Cache Instances, since the Lost Message state does not survive across process termination.)

Clearing PubSub+ Cache Instance Statistics

To clear the statistics for one or more PubSub+ Cache Instances under a Distributed Cache, enter the following Privileged EXEC command:

solace# clear cache-instance <name> [cache-cluster <cluster-name>] [distributed-cache <cache-name>] [message-vpn <vpn-name>] stats

Where:

<name> is the PubSub+ Cache Instance name

<cluster-name> is the name of the Cache Cluster that the PubSub+ Cache Instance belongs to

<cache-name> is the name of the Distributed Cache that the PubSub+ Cache Instance belongs to

<vpn-name> is the name of the Message VPN that the PubSub+ Cache Instance belongs to

Provide feedback