Solace recommends monitoring the events in the system Syslog (system.log or system facility) as it contains all the events relevant to the health of the event broker. These events are grouped into seven categories and are listed in Solace System Event Categories.
For detailed information on the events generated by an event broker, such as the message, severity level, recommended actions, and message format, refer to the Syslog Event Reference.
The System Events Recommended for Monitoring and Message VPN Events Recommended for Monitoring tables shown below list the minimum set of events that are recommended for monitoring by management applications. The objective of these lists is to serve as a starting point for you when integrating with PubSub+. Additional events may need to be monitored depending on your application's use-case, and any additional event broker features you decide to use. These events indicate issues with event broker components, or thresholds being reached for system resources, and help identify issues in the SYSTEM and VPN categories. Some events also have a corresponding “Clear" event, which is an event that indicates that a condition has cleared.
The actual list of events to be monitored depends on the capabilities used by the application. For example, if LDAP isn't used as the authentication mechanism, then the corresponding events don't need to be monitored.
System Events Recommended for Monitoring
| Set Event | Severity | Clear Event | Severity |
|---|---|---|---|
| SYSTEM_AD_DELIVERED_UNACKED_MSGS_HIGH | WARN | SYSTEM_AD_DELIVERED_UNACKED_MSGS_HIGH_CLEAR | INFO |
| SYSTEM_AD_DELIVERED_UNACKED_MSGS_EXCEED | |||
| SYSTEM_AD_DISK_USAGE_HIGH | WARN | SYSTEM_AD_DISK_USAGE_HIGH_CLEAR | INFO |
| SYSTEM_AD_DISK_USAGE_EXCEEDED | |||
| SYSTEM_AD_EGRESS_FLOWS_HIGH | WARN | SYSTEM_AD_EGRESS_FLOWS_HIGH_CLEAR | INFO |
| SYSTEM_AD_MAX_EGRESS_FLOWS_EXCEEDED | |||
| SYSTEM_AD_ENDPOINTS_HIGH | WARN | SYSTEM_AD_ENDPOINTS_HIGH_CLEAR | INFO |
| SYSTEM_AD_MAX_ENDPOINTS_EXCEEDED | |||
| SYSTEM_AD_INGRESS_FLOWS_HIGH | WARN | SYSTEM_AD_INGRESS_FLOWS_HIGH_CLEAR | INFO |
| SYSTEM_AD_MAX_INGRESS_FLOWS_EXCEEDED | |||
| SYSTEM_AD_MSG_SPOOL_HIGH | WARN | SYSTEM_AD_MSG_SPOOL_HIGH_CLEAR | INFO |
| SYSTEM_AD_MSG_SPOOL_QUOTA_EXCEED | |||
| SYSTEM_AD_MSG_COUNT_UTILIZATION_HIGH | WARN | SYSTEM_AD_MSG_COUNT_UTILIZATION_HIGH_CLEAR | INFO |
| SYSTEM_AD_MSG_COUNT_UTILIZATION_EXCEEDED | |||
| SYSTEM_AD_FLASH_FAILED | ERROR | ||
| SYSTEM_AD_RESTORE_FAILED | ERROR | ||
| SYSTEM_AD_SPOOL_FILES_HIGH SYSTEM_AD_SPOOL_FILES_EXCEEDED | WARN | SYSTEM_AD_SPOOL_FILES_HIGH_CLEAR | INFO |
| SYSTEM_AD_TRANSACTIONS_HIGH | WARN | SYSTEM_AD_TRANSACTIONS_HIGH_CLEAR | INFO |
| SYSTEM_AD_TRANSACTIONS_EXCEED | |||
| SYSTEM_AD_TRANSACTED_SESSION_RESOURCE_UTILIZATION_HIGH | WARN | SYSTEM_AD_TRANSACTED_SESSION_RESOURCE_UTILIZATION_HIGH_CLEAR | INFO |
| SYSTEM_AD_TRANSACTED_SESSION_RESOURCE_UTILIZATION_EXCEEDED | |||
| SYSTEM_AD_MSG_SPOOL_CHG | NOTICE | ||
| SYSTEM_AD_TRANSACTED_SESSIONS_HIGH | WARN | SYSTEM_AD_TRANSACTED_SESSIONS_HIGH_CLEAR | INFO |
| SYSTEM_AD_TRANSACTED_SESSIONS_EXCEED | |||
| SYSTEM_ADB_CRITICAL_HARDWARE_FAILURE | CRIT | ||
| SYSTEM_AUTHENTICATION_ADMIN_BIND_FAIL | INFO | ||
| SYSTEM_AUTHENTICATION_ADMIN_CONN_DOWN | INFO | SYSTEM_AUTHENTICATION_ADMIN_CONN_UP | INFO |
| SYSTEM_AUTHENTICATION_BIND_CONN_DOWN | INFO | SYSTEM_AUTHENTICATION_BIND_CONN_UP | INFO |
| SYSTEM_AUTHENTICATION_CRL_DOWNLOAD_FAILED | WARN | SYSTEM_AUTHENTICATION_CRL_DOWNLOAD_SUCCESS | INFO |
| SYSTEM_AUTHENTICATION_SESSION_DENIED | NOTICE | ||
| SYSTEM_AUTHENTICATION_SHELL_ACCESS_DENIED | NOTICE | ||
| SYSTEM_AUTHENTICATION_TLS_START_FAIL | WARN | ||
| SYSTEM_CFGSYNC_DOWN | WARN | SYSTEM_CFGSYNC_UP | INFO |
| SYSTEM_CHASSIS_AVAILABLE_STORAGE_LOW | WARN | SYSTEM_CHASSIS_AVAILABLE_STORAGE_LOW_CLEAR | INFO |
| SYSTEM_CHASSIS_BLADE_DOWN | CRIT | SYSTEM_CHASSIS_BLADE_UP | INFO |
| SYSTEM_CHASSIS_BLADE_POST_CRITICAL_FAILURE | CRIT | ||
| SYSTEM_CHASSIS_BLADE_POST_NONCRITICAL_NOTIFICATION | WARN | ||
| SYSTEM_CHASSIS_BOOT_DISK_FAIL | CRIT | ||
| SYSTEM_CHASSIS_DISK_DOWN | WARN | SYSTEM_CHASSIS_DISK_UP | INFO |
| SYSTEM_CHASSIS_DISK_UTILIZATION_HIGH | WARN | SYSTEM_CHASSIS_DISK_UTILIZATION_HIGH_CLEAR | INFO |
| SYSTEM_CHASSIS_DISK_UTILIZATION_MAX | |||
| SYSTEM_CHASSIS_FABRIC_POST_CRITICAL_FAILURE | CRIT | ||
| SYSTEM_CHASSIS_FABRIC_POST_NONCRITICAL_NOTIFICATION | WARN | ||
| SYSTEM_CHASSIS_FAN_HIGH | WARN | SYSTEM_CHASSIS_FAN_HIGH_CLEAR | INFO |
| SYSTEM_CHASSIS_FAN_LOW | WARN | SYSTEM_CHASSIS_FAN_LOW_CLEAR | INFO |
| SYSTEM_CHASSIS_HARDWARE_SOFT_ERROR | ERROR | ||
| SYSTEM_CHASSIS_INODE_UTILIZATION_HIGH | WARN | SYSTEM_CHASSIS_INODE_UTILIZATION_HIGH_CLEAR | INFO |
| SYSTEM_CHASSIS_INODE_UTILIZATION_MAX | |||
| SYSTEM_CHASSIS_PHYSICAL_CPU_FAIL | CRIT | ||
| SYSTEM_CHASSIS_PHYSICAL_MEMORY_FAIL | CRIT | ||
| SYSTEM_CHASSIS_POST_CRITICAL_FAILURE | CRIT | ||
| SYSTEM_CHASSIS_POST_NONCRITICAL_NOTIFICATION | WARN | ||
| SYSTEM_CHASSIS_POWER_MODULE_DOWN | WARN | SYSTEM_CHASSIS_POWER_MODULE_UP | INFO |
| SYSTEM_CHASSIS_TEMPERATURE_HIGH | WARN | SYSTEM_CHASSIS_TEMPERATURE_HIGH_CLEAR | INFO |
| SYSTEM_CHASSIS_TEMPERATURE_LOW | WARN | SYSTEM_CHASSIS_TEMPERATURE_LOW_CLEAR | INFO |
| SYSTEM_CHASSIS_UNSUPPORTED_BLADE_CONFIGURATION | WARN | ||
| SYSTEM_CHASSIS_UNSUPPORTED_CONFIGURATION | WARN | ||
| SYSTEM_CHASSIS_UNSUPPORTED_DISK_CONFIGURATION | WARN | SYSTEM_CHASSIS_UNSUPPORTED_DISK_CONFIGURATION_CLEAR | INFO |
| SYSTEM_CHASSIS_VOLTAGE_HIGH | WARN | SYSTEM_CHASSIS_VOLTAGE_HIGH_CLEAR | INFO |
| SYSTEM_CHASSIS_VOLTAGE_LOW | WARN | SYSTEM_CHASSIS_VOLTAGE_LOW_CLEAR | INFO |
| SYSTEM_CLIENT_CONNECTIONS_HIGH | WARN | SYSTEM_CLIENT_CONNECTIONS_HIGH_CLEAR | INFO |
| SYSTEM_CLIENT_CONNECTIONS_EXCEEDED | |||
| SYSTEM_CLIENT_EG_MSG_RATE_HIGH | WARN | SYSTEM_CLIENT_EG_MSG_RATE_HIGH_CLEAR | INFO |
| SYSTEM_CLIENT_ING_MSG_RATE_HIGH | WARN | SYSTEM_CLIENT_ING_MSG_RATE_HIGH_CLEAR | INFO |
| SYSTEM_CLIENT_SUBSCRIPTIONS_HIGH | WARN | SYSTEM_CLIENT_SUBSCRIPTIONS_HIGH_CLEAR | INFO |
| SYSTEM_CLIENT_SUBSCRIPTIONS_MEMORY_HIGH | WARN | SYSTEM_CLIENT_SUBSCRIPTIONS_MEMORY_HIGH_CLEAR | INFO |
| SYSTEM_DNS_NAME_SERVER_DOWN | WARN | SYSTEM_DNS_NAME_SERVER_UP | INFO |
| SYSTEM_HA_DISK_PATH_CHG | NOTICE | ||
| SYSTEM_HA_REDUN_GROUP_NODE_JOINED | WARN | ||
| SYSTEM_HA_REDUN_GROUP_NODE_LEFT | WARN | ||
| SYSTEM_HA_REDUN_STATE_DOWN | WARN | SYSTEM_HA_REDUN_STATE_UP | INFO |
| SYSTEM_HA_TIME_SKEW | WARN | SYSTEM_HA_TIME_SKEW_CLEARED | INFO |
| SYSTEM_HA_VRRP_STATE_DOWN | WARN | SYSTEM_HA_VRRP_STATE_UP | INFO |
| SYSTEM_HA_VR_STATE_STANDBY | INFO | SYSTEM_HA_VR_STATE_ACTIVE | INFO |
| SYSTEM_LINK_ADB_HELLO_PROTOCOL_DOWN | ERROR | SYSTEM_LINK_ADB_HELLO_PROTOCOL_UP | INFO |
| SYSTEM_LINK_ADB_LINK_DOWN | WARN | SYSTEM_LINK_ADB_LINK_UP | INFO |
| SYSTEM_LINK_ETH_LINK_DOWN | WARN | SYSTEM_LINK_ETH_LINK_UP | INFO |
| SYSTEM_LINK_HBA_LINK_DOWN | WARN | SYSTEM_LINK_HBA_LINK_UP | INFO |
| SYSTEM_LINK_IP_ADDRESS_CHANGED | INFO | ||
| SYSTEM_LINK_LAG_DOWN | WARN | SYSTEM_LINK_LAG_UP | INFO |
| SYSTEM_LINK_PATH_TO_DISK_ARRAY_DOWN | ERROR | SYSTEM_LINK_PATH_TO_DISK_ARRAY_UP | INFO |
| SYSTEM_LOGGING_LOST_EVENTS | WARN | ||
| SYSTEM_NAB_BUFFER_LOAD_FACTOR_HIGH | WARN | SYSTEM_NAB_BUFFER_LOAD_FACTOR_HIGH_CLEAR | INFO |
| SYSTEM_NAB_CRITICAL_HARDWARE_FAILURE | CRIT | ||
| SYSTEM_NAB_CRITICAL_HARDWARE_SOFT_FAILURE | CRIT | ||
| SYSTEM_NAB_NONCRITICAL_HARDWARE_NOTIFICATION | WARN | ||
| SYSTEM_NAB_NONCRITICAL_HARDWARE_SOFT_FAILURE | NOTICE | ||
| SYSTEM_NAB_ZIP_ENGINE_RESTART_INITIATED | NOTICE | SYSTEM_NAB_ZIP_ENGINE_RESTART_COMPLETE | NOTICE |
| SYSTEM_NTP_SERVER_DOWN | WARN | SYSTEM_NTP_SERVER_UP | INFO |
| SYSTEM_SERVICE_LISTEN_PORT_DISABLE | NOTICE | SYSTEM_SERVICE_LISTEN_PORT_ENABLE | NOTICE |
| SYSTEM_SSL_CONNECTIONS_HIGH | WARN | SYSTEM_SSL_CONNECTIONS_HIGH_CLEAR | INFO |
| SYSTEM_SSL_CONNECTIONS_EXCEEDED | |||
| SYSTEM_SYSTEM_STARTUP_COMPLETE | WARN |
Message VPN Events Recommended for Monitoring
| Set Event | Severity | Clear Event | Severity |
|---|---|---|---|
| VPN_AD_BIND_COUNT_HIGH | WARN | VPN_AD_BIND_COUNT_HIGH_CLEAR | INFO |
| VPN_AD_CLIENT_USERNAME_ENDPOINTS_HIGH | WARN | VPN_AD_CLIENT_USERNAME_ENDPOINTS_HIGH_CLEAR | INFO |
| VPN_AD_CLIENT_USERNAME_MAX_ENDPOINTS_EXCEEDED | |||
| VPN_AD_EGRESS_FLOWS_HIGH | WARN | VPN_AD_EGRESS_FLOWS_HIGH_CLEAR | INFO |
| VPN_AD_MAX_EGRESS_FLOWS_EXCEEDED | |||
| VPN_AD_ENDPOINTS_HIGH | WARN | VPN_AD_ENDPOINTS_HIGH_CLEAR | INFO |
| VPN_AD_MAX_ENDPOINTS_EXCEEDED | |||
| VPN_AD_INGRESS_FLOWS_HIGH | WARN | VPN_AD_INGRESS_FLOWS_HIGH_CLEAR | INFO |
| VPN_AD_MAX_INGRESS_FLOWS_EXCEEDED | |||
| VPN_AD_MSG_SPOOL_HIGH | WARN | VPN_AD_MSG_SPOOL_HIGH_CLEAR | INFO |
| VPN_AD_MSG_SPOOL_QUOTA_EXCEED | |||
| VPN_AD_TRANSACTED_SESSIONS_HIGH | WARN | VPN_AD_TRANSACTED_SESSIONS_HIGH_CLEAR | INFO |
| VPN_AD_TRANSACTED_SESSIONS_EXCEED | |||
| VPN_AD_TRANSACTIONS_HIGH | WARN | VPN_AD_TRANSACTIONS_HIGH_CLEAR | INFO |
| VPN_AD_TRANSACTIONS_EXCEED | |||
| VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_HIGH | WARN | VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_HIGH_CLEAR | INFO |
| VPN_AD_MSG_SPOOL_REJECT_LOW_PRIORITY_MSG_LIMIT_EXCEED | |||
| VPN_BRIDGING_LINK_DOWN | WARN | VPN_BRIDGING_LINK_UP | INFO |
| VPN_BRIDGING_LINK_TTL_EXCEEDED | WARN | VPN_BRIDGING_LINK_TTL_EXCEEDED_CLEAR | INFO |
| VPN_BRIDGING_BRIDGE_STALLED | WARN | ||
| VPN_BRIDGING_LINK_REJECTED | NOTICE | ||
| VPN_BRIDGING_LINK_CONFIG_MISMATCH | WARN | ||
| VPN_CLIENT_USERNAME_CONNECTIONS_HIGH | WARN | VPN_CLIENT_USERNAME_CONNECTIONS_HIGH_CLEAR | INFO |
| VPN_CLIENT_USERNAME_CONNECTIONS_EXCEEDED | WARN | ||
| VPN_REPLICATION_SERVICE_DEGRADED | WARN | VPN_REPLICATION_SERVICE_DEGRADED_CLEAR | INFO |
| VPN_SERVICE_LISTEN_PORT_STATE_CHANGE | WARN | ||
| VPN_SERVICE_SMF_STATE_CHANGE | WARN | ||
| VPN_VPN_CONNECTIONS_HIGH | WARN | VPN_VPN_CONNECTIONS_HIGH_CLEAR | INFO |
| VPN_VPN_MAX_CONNECTIONS_EXCEEDED | |||
| VPN_VPN_EG_MSG_RATE_HIGH | WARN | VPN_VPN_EG_MSG_RATE_HIGH_CLEAR | INFO |
| VPN_VPN_ING_MSG_RATE_HIGH | WARN | VPN_VPN_ING_MSG_RATE_HIGH_CLEAR | INFO |
| VPN_VPN_SUBSCRIPTIONS_HIGH | WARN | VPN_VPN_SUBSCRIPTIONS_HIGH_CLEAR | INFO |
| VPN_VPN_MAX_SUBSCRIPTIONS_EXCEEDED | |||
| VPN_VPN_STATE_CHANGE | WARN |
Solace System Event Categories
The system level events generated by the event broker can be grouped into the categories noted in the following subsections.
Chassis Syslog messages: SYSTEM_CHASSIS_*
These event messages generally indicate a hardware failure that requires investigation and may result in a degraded system.
High Availability Syslog Messages: SYSTEM_HA_*
These event messages indicate a redundancy state change. Investigation is required to understand why they occurred. For example, planned maintenance on the mate event broker, or an unplanned event.
Link Syslog Messages: SYSTEM_LINK_*
These event messages indicate a link state change, either to the mate event broker, the external disk storage array, or the network. Investigation is required to understand why they occurred.
Both SYSTEM_HA_* and SYSTEM_LINK_* event messages can be useful to detect a mate event broker failure where the failing event broker doesn't have the ability to raise an event itself for the failure (for example, power failure).
NAB-related Syslog Messages: SYSTEM_NAB_*
These event messages indicate component failures on the NAB as well as when NAB buffers are exhausted to the point clients may have been disconnected.
Solace recommends you report any NAB component failures for further investigation by contacting Solace.
Routing Syslog Messages: SYSTEM_ROUTING_*
If Multiple-Node Routing is being used, these event messages indicate routing state changes, including potential problems.
SYSTEM_SERVICE_LISTEN_PORT_ENABLE and SYSTEM_SYSTEM_STARTUP_COMPLETE
These event messages indicated that the firmware on the appliance has completed its initialization and can be used to detect that an event broker has restarted.
Message VPN Bridging Syslog Messages: VPN_BRIDGING_*
If Message VPN bridging is being used, these event messages indicate bridge state changes, including potential problems.