Minimum Recommended Events for Monitoring

It's recommended to monitor the events in the system Syslog (system.log or system facility) as it contains all the events relevant to the health of the event broker. These events are grouped into seven categories and are listed in Solace System Event Categories.

For detailed information on the events generated by the event broker, such as the message, severity level, recommended actions, and message format, refer to the Syslog Event Reference.

The System Events Recommended for Monitoring and Message VPN Events Recommended for Monitoring tables shown below list the minimum set of events that are recommended for monitoring by management applications. The objective of these lists is to serve as a starting point for you when integrating with Solace PubSub+. Additional events may need to be monitored depending on your application's use-case, and any additional event broker features you decide to use. These events indicate issues with event broker components, or thresholds being reached for system resources, and help identify issues in the SYSTEM and VPN categories. Some events also have a corresponding “Clear" event, which is an event that indicates that a condition has cleared.

The actual list of events to be monitored depends on the capabilities used by the application. For example, if PubSub+ Cache isn't being used, or LDAP isn't used as the authentication mechanism, then the corresponding events don't need to be monitored.

System Events Recommended for Monitoring

No. Set Event Severity Clear Event Severity
1 SYSTEM_AD_DELIVERED_UNACKED_MSGS_HIGH WARN SYSTEM_AD_DELIVERED_UNACKED_MSGS_HIGH_CLEAR INFO
SYSTEM_AD_DELIVERED_UNACKED_MSGS_EXCEED
2 SYSTEM_AD_DISK_USAGE_HIGH WARN SYSTEM_AD_DISK_USAGE_HIGH_CLEAR INFO
SYSTEM_AD_DISK_USAGE_EXCEEDED
3 SYSTEM_AD_EGRESS_FLOWS_HIGH WARN SYSTEM_AD_EGRESS_FLOWS_HIGH_CLEAR INFO
SYSTEM_AD_MAX_EGRESS_FLOWS_EXCEEDED
4 SYSTEM_AD_ENDPOINTS_HIGH WARN SYSTEM_AD_ENDPOINTS_HIGH_CLEAR INFO
SYSTEM_AD_MAX_ENDPOINTS_EXCEEDED
5 SYSTEM_AD_INGRESS_FLOWS_HIGH WARN SYSTEM_AD_INGRESS_FLOWS_HIGH_CLEAR INFO
SYSTEM_AD_MAX_INGRESS_FLOWS_EXCEEDED
6 SYSTEM_AD_MSG_CACHE_USAGE_HIGH WARN SYSTEM_AD_MSG_CACHE_USAGE_HIGH_CLEAR INFO
SYSTEM_AD_MAX_MSG_CACHE_USAGE_EXCEEDED
7 SYSTEM_AD_MSG_SPOOL_HIGH WARN SYSTEM_AD_MSG_SPOOL_HIGH_CLEAR INFO
SYSTEM_AD_MSG_SPOOL_QUOTA_EXCEED
8 SYSTEM_AD_MSG_COUNT_UTILIZATION_HIGH WARN SYSTEM_AD_MSG_COUNT_UTILIZATION_HIGH_CLEAR INFO
SYSTEM_AD_MSG_COUNT_UTILIZATION_EXCEEDED
9 SYSTEM_AD_FLASH_FAILED ERROR    
10 SYSTEM_AD_RESTORE_FAILED ERROR    
11 SYSTEM_AD_SPOOL_FILES_HIGH SYSTEM_AD_SPOOL_FILES_EXCEEDED WARN SYSTEM_AD_SPOOL_FILES_HIGH_CLEAR INFO
12 SYSTEM_AD_TRANSACTIONS_HIGH WARN SYSTEM_AD_TRANSACTIONS_HIGH_CLEAR INFO
SYSTEM_AD_TRANSACTIONS_EXCEED
13 SYSTEM_AD_TRANSACTED_SESSION_RESOURCE_UTILIZATION_HIGH WARN SYSTEM_AD_TRANSACTED_SESSION_RESOURCE_UTILIZATION_HIGH_CLEAR INFO
SYSTEM_AD_TRANSACTED_SESSION_RESOURCE_UTILIZATION_EXCEEDED
14 SYSTEM_AD_MSG_SPOOL_CHG NOTICE    
15 SYSTEM_AD_TRANSACTED_SESSIONS_HIGH WARN SYSTEM_AD_TRANSACTED_SESSIONS_HIGH_CLEAR INFO
SYSTEM_AD_TRANSACTED_SESSIONS_EXCEED
16 SYSTEM_ADB_CRITICAL_HARDWARE_FAILURE CRIT    
17 SYSTEM_AUTHENTICATION_ADMIN_BIND_FAIL INFO    
18 SYSTEM_AUTHENTICATION_ADMIN_CONN_DOWN INFO SYSTEM_AUTHENTICATION_ADMIN_CONN_UP INFO
19 SYSTEM_AUTHENTICATION_BIND_CONN_DOWN INFO SYSTEM_AUTHENTICATION_BIND_CONN_UP INFO
20 SYSTEM_AUTHENTICATION_CRL_DOWNLOAD_FAILED WARN SYSTEM_AUTHENTICATION_CRL_DOWNLOAD_SUCCESS INFO
21 SYSTEM_AUTHENTICATION_SESSION_DENIED NOTICE    
22 SYSTEM_AUTHENTICATION_SHELL_ACCESS_DENIED NOTICE    
23 SYSTEM_AUTHENTICATION_TLS_START_FAIL WARN    
24 SYSTEM_CFGSYNC_DOWN WARN SYSTEM_CFGSYNC_UP INFO
25 SYSTEM_CHASSIS_AVAILABLE_STORAGE_LOW WARN SYSTEM_CHASSIS_AVAILABLE_STORAGE_LOW_CLEAR INFO
26 SYSTEM_CHASSIS_BLADE_DOWN CRIT SYSTEM_CHASSIS_BLADE_UP INFO
27 SYSTEM_CHASSIS_BLADE_POST_CRITICAL_FAILURE CRIT    
28 SYSTEM_CHASSIS_BLADE_POST_NONCRITICAL_NOTIFICATION WARN    
29 SYSTEM_CHASSIS_BOOT_DISK_FAIL CRIT    
30 SYSTEM_CHASSIS_DISK_DOWN WARN SYSTEM_CHASSIS_DISK_UP INFO
31 SYSTEM_CHASSIS_DISK_UTILIZATION_HIGH WARN SYSTEM_CHASSIS_DISK_UTILIZATION_HIGH_CLEAR INFO
SYSTEM_CHASSIS_DISK_UTILIZATION_MAX
32 SYSTEM_CHASSIS_FABRIC_POST_CRITICAL_FAILURE CRIT    
33 SYSTEM_CHASSIS_FABRIC_POST_NONCRITICAL_NOTIFICATION WARN    
34 SYSTEM_CHASSIS_FAN_HIGH WARN SYSTEM_CHASSIS_FAN_HIGH_CLEAR INFO
35 SYSTEM_CHASSIS_FAN_LOW WARN SYSTEM_CHASSIS_FAN_LOW_CLEAR INFO
36 SYSTEM_CHASSIS_HARDWARE_SOFT_ERROR ERROR    
37 SYSTEM_CHASSIS_INODE_UTILIZATION_HIGH WARN SYSTEM_CHASSIS_INODE_UTILIZATION_HIGH_CLEAR INFO
SYSTEM_CHASSIS_INODE_UTILIZATION_MAX
38 SYSTEM_CHASSIS_PHYSICAL_CPU_FAIL CRIT    
39 SYSTEM_CHASSIS_PHYSICAL_MEMORY_FAIL CRIT    
40 SYSTEM_CHASSIS_POST_CRITICAL_FAILURE CRIT    
41 SYSTEM_CHASSIS_POST_NONCRITICAL_NOTIFICATION WARN    
42 SYSTEM_CHASSIS_POWER_MODULE_DOWN WARN SYSTEM_CHASSIS_POWER_MODULE_UP INFO
43 SYSTEM_CHASSIS_TEMPERATURE_HIGH WARN SYSTEM_CHASSIS_TEMPERATURE_HIGH_CLEAR INFO
44 SYSTEM_CHASSIS_TEMPERATURE_LOW WARN SYSTEM_CHASSIS_TEMPERATURE_LOW_CLEAR INFO
45 SYSTEM_CHASSIS_UNSUPPORTED_BLADE_CONFIGURATION WARN    
46 SYSTEM_CHASSIS_UNSUPPORTED_CONFIGURATION WARN    
47 SYSTEM_CHASSIS_UNSUPPORTED_DISK_CONFIGURATION WARN SYSTEM_CHASSIS_UNSUPPORTED_DISK_CONFIGURATION_CLEAR INFO
48 SYSTEM_CHASSIS_VOLTAGE_HIGH WARN SYSTEM_CHASSIS_VOLTAGE_HIGH_CLEAR INFO
49 SYSTEM_CHASSIS_VOLTAGE_LOW WARN SYSTEM_CHASSIS_VOLTAGE_LOW_CLEAR INFO
50 SYSTEM_CLIENT_CONNECTIONS_HIGH WARN SYSTEM_CLIENT_CONNECTIONS_HIGH_CLEAR INFO
SYSTEM_CLIENT_CONNECTIONS_EXCEEDED
51 SYSTEM_CLIENT_EG_MSG_RATE_HIGH WARN SYSTEM_CLIENT_EG_MSG_RATE_HIGH_CLEAR INFO
52 SYSTEM_CLIENT_ING_MSG_RATE_HIGH WARN SYSTEM_CLIENT_ING_MSG_RATE_HIGH_CLEAR INFO
53 SYSTEM_CLIENT_SUBSCRIPTIONS_HIGH WARN SYSTEM_CLIENT_SUBSCRIPTIONS_HIGH_CLEAR INFO
54 SYSTEM_CLIENT_SUBSCRIPTIONS_MEMORY_HIGH WARN SYSTEM_CLIENT_SUBSCRIPTIONS_MEMORY_HIGH_CLEAR INFO
55 SYSTEM_DNS_NAME_SERVER_DOWN WARN SYSTEM_DNS_NAME_SERVER_UP INFO
56 SYSTEM_HA_DISK_PATH_CHG NOTICE    
57 SYSTEM_HA_REDUN_GROUP_NODE_JOINED WARN    
58 SYSTEM_HA_REDUN_GROUP_NODE_LEFT WARN    
59 SYSTEM_HA_REDUN_STATE_DOWN WARN SYSTEM_HA_REDUN_STATE_UP INFO
60 SYSTEM_HA_TIME_SKEW WARN SYSTEM_HA_TIME_SKEW_CLEARED INFO
61 SYSTEM_HA_VRRP_STATE_DOWN WARN SYSTEM_HA_VRRP_STATE_UP INFO
62 SYSTEM_HA_VR_STATE_STANDBY INFO SYSTEM_HA_VR_STATE_ACTIVE INFO
63 SYSTEM_LINK_ADB_HELLO_PROTOCOL_DOWN ERROR SYSTEM_LINK_ADB_HELLO_PROTOCOL_UP INFO
64 SYSTEM_LINK_ADB_LINK_DOWN WARN SYSTEM_LINK_ADB_LINK_UP INFO
65 SYSTEM_LINK_ETH_LINK_DOWN WARN SYSTEM_LINK_ETH_LINK_UP INFO
66 SYSTEM_LINK_HBA_LINK_DOWN WARN SYSTEM_LINK_HBA_LINK_UP INFO
67 SYSTEM_LINK_IP_ADDRESS_CHANGED INFO    
68 SYSTEM_LINK_LAG_DOWN WARN SYSTEM_LINK_LAG_UP INFO
69 SYSTEM_LINK_PATH_TO_DISK_ARRAY_DOWN ERROR SYSTEM_LINK_PATH_TO_DISK_ARRAY_UP INFO
70 SYSTEM_LOGGING_LOST_EVENTS WARN    
71 SYSTEM_NAB_BUFFER_LOAD_FACTOR_HIGH WARN SYSTEM_NAB_BUFFER_LOAD_FACTOR_HIGH_CLEAR INFO
72 SYSTEM_NAB_CRITICAL_HARDWARE_FAILURE CRIT    
73 SYSTEM_NAB_CRITICAL_HARDWARE_SOFT_FAILURE CRIT    
74 SYSTEM_NAB_NONCRITICAL_HARDWARE_NOTIFICATION WARN    
75 SYSTEM_NAB_NONCRITICAL_HARDWARE_SOFT_FAILURE NOTICE    
76 SYSTEM_NAB_ZIP_ENGINE_RESTART_INTIATED NOTICE SYSTEM_NAB_ZIP_ENGINE_RESTART_COMPLETE NOTICE
77 SYSTEM_NTP_SERVER_DOWN WARN SYSTEM_NTP_SERVER_UP INFO
78 SYSTEM_SERVICE_LISTEN_PORT_DISABLE NOTICE SYSTEM_SERVICE_LISTEN_PORT_ENABLE NOTICE
79 SYSTEM_SSL_CONNECTIONS_HIGH WARN SYSTEM_SSL_CONNECTIONS_HIGH_CLEAR INFO
SYSTEM_SSL_CONNECTIONS_EXCEEDED
80 SYSTEM_SYSTEM_STARTUP_COMPLETE WARN    

Message VPN Events Recommended for Monitoring

No. Set Event Severity Clear Event Severity
81 VPN_AD_BIND_COUNT_HIGH WARN VPN_AD_BIND_COUNT_HIGH_CLEAR INFO
82 VPN_AD_CLIENT_USERNAME_ENDPOINTS_HIGH WARN VPN_AD_CLIENT_USERNAME_ENDPOINTS_HIGH_CLEAR INFO
VPN_AD_CLIENT_USERNAME_MAX_ENDPOINTS_EXCEEDED
83 VPN_AD_EGRESS_FLOWS_HIGH WARN VPN_AD_EGRESS_FLOWS_HIGH_CLEAR INFO
VPN_AD_MAX_EGRESS_FLOWS_EXCEEDED  
84 VPN_AD_ENDPOINTS_HIGH WARN VPN_AD_ENDPOINTS_HIGH_CLEAR INFO
VPN_AD_MAX_ENDPOINTS_EXCEEDED
85 VPN_AD_INGRESS_FLOWS_HIGH WARN VPN_AD_INGRESS_FLOWS_HIGH_CLEAR INFO
VPN_AD_MAX_INGRESS_FLOWS_EXCEEDED
86 VPN_AD_MSG_SPOOL_HIGH WARN VPN_AD_MSG_SPOOL_HIGH_CLEAR INFO
VPN_AD_MSG_SPOOL_QUOTA_EXCEED
87 VPN_AD_TRANSACTED_SESSIONS_HIGH WARN VPN_AD_TRANSACTED_SESSIONS_HIGH_CLEAR INFO
VPN_AD_TRANSACTED_SESSIONS_EXCEED
88 VPN_AD_TRANSACTIONS_HIGH WARN VPN_AD_TRANSACTIONS_HIGH_CLEAR INFO
VPN_AD_TRANSACTIONS_EXCEED
89 VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_HIGH WARN VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_HIGH_CLEAR INFO
VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_EXCEED
90 VPN_BRIDGING_LINK_DOWN WARN VPN_BRIDGING_LINK_UP INFO
91 VPN_BRIDGING_LINK_TTL_EXCEEDED WARN VPN_BRIDGING_LINK_TTL_EXCEEDED_CLEAR INFO
92 VPN_BRIDGING_BRIDGE_STALLED WARN    
93 VPN_BRIDGING_LINK_REJECTED NOTICE    
94 VPN_BRIDGING_LINK_CONFIG_MISMATCH WARN    
95 VPN_CLIENT_USERNAME_CONNECTIONS_HIGH WARN VPN_CLIENT_USERNAME_CONNECTIONS_HIGH_CLEAR INFO
96 VPN_CLIENT_USERNAME_CONNECTIONS_EXCEEDED WARN    
97 VPN_REPLICATION_SERVICE_DEGRADED WARN VPN_REPLICATION_SERVICE_DEGRADED_CLEAR INFO
98 VPN_SERVICE_LISTEN_PORT_STATE_CHANGE WARN    
99 VPN_SERVICE_SMF_STATE_CHANGE WARN    
100 VPN_VPN_CONNECTIONS_HIGH WARN VPN_VPN_CONNECTIONS_HIGH_CLEAR INFO
101 VPN_VPN_MAX_CONNECTIONS_EXCEEDED
102 VPN_VPN_EG_MSG_RATE_HIGH WARN VPN_VPN_EG_MSG_RATE_HIGH_CLEAR INFO
103 VPN_VPN_ING_MSG_RATE_HIGH WARN VPN_VPN_ING_MSG_RATE_HIGH_CLEAR INFO
104 VPN_VPN_SUBSCRIPTIONS_HIGH WARN VPN_VPN_SUBSCRIPTIONS_HIGH_CLEAR INFO
VPN_VPN_MAX_SUBSCRIPTIONS_EXCEEDED
105 VPN_VPN_STATE_CHANGE WARN    

Solace System Event Categories

The system level events generated by the event broker can be grouped into the categories noted in the following subsections.

Chassis Syslog messages: SYSTEM_CHASSIS_*

These event messages generally indicate a hardware failure that requires investigation and may result in a degraded system.

High Availability Syslog Messages: SYSTEM_HA_*

These event messages indicate a redundancy state change. Investigation is required to understand why they occurred. For example, planned maintenance on the mate event broker, or an unplanned event.

Link Syslog Messages: SYSTEM_LINK_*

These event messages indicate a link state change, either to the mate event broker, the external disk storage array, or the network. Investigation is required to understand why they occurred.

Both SYSTEM_HA_* and SYSTEM_LINK_* event messages can be useful to detect a mate event broker failure where the failing event broker doesn't have the ability to raise an event itself for the failure (for example, power failure).

NAB-related Syslog Messages: SYSTEM_NAB_*

These event messages indicate component failures on the NAB as well as when NAB buffers are exhausted to the point clients may have been disconnected.

Solace recommends you report any NAB component failures for further investigation by contacting Solace.

Routing Syslog Messages: SYSTEM_ROUTING_*

If Multiple-Node Routing is being used, these event messages indicate routing state changes, including potential problems.

SYSTEM_SERVICE_LISTEN_PORT_ENABLE and SYSTEM_SYSTEM_STARTUP_COMPLETE

These event messages indicated that the firmware on the appliance has completed its initialization and can be used to detect that an event broker has restarted.

Message VPN Bridging Syslog Messages: VPN_BRIDGING_*

If Message VPN bridging is being used, these event messages indicate bridge state changes, including potential problems.