Monitoring Events

Solace PubSub+ event brokers generate events to notify monitoring applications of state changes. These events include:

  • Client Events - such as connects, disconnects, binds, flow events, etc.
  • VPN Events - such PubSub+ Cache state changes, endpoint creation, quotas exceeded, etc.
  • System Events - such as Multi-Node Routing events, power module up/down, fan speed changes, etc.

Solace PubSub+ event brokers also allow for auditing administrative commands via Solace CLI or SEMP.

The following guidelines should be used for handling events:

  • Critical events should be escalated by contacting Solace.
  • Applications can subscribe to a well-known topic to receive event logs published by an event broker. Typically, standard applications wouldn't subscribe to these, but management applications might. For example, a watchdog for a group of applications. Refer to Event Publishing Over the Message Bus for more details.
  • Applications can also request information using SEMP over HTTP or the message bus for a snapshot of monitoring details and statistics. Refer to Request / Reply Monitoring Using SEMP for more information.
  • To ensure that log timestamps are synchronized amongst multiple event brokers, including those in HA pairs, all event brokers should be configured to use NTP to sync clocks and ensure consistent log timestamps.

Event Publishing Over the Message Bus

Event brokers can be configured to publish events that provide management and status information to well-known topics onto the message bus. The events can be of the following type:

  • System-level events - such as, fan failure, redundancy down
  • Message VPN-level events - such as, spool usage exceeded, ingress/egress message rates exceeded
  • Client events - such as, egress discard, client bind/unbind
  • Subscription events - such as, subscription add/remove

To be able to receive events published over the message bus, publication of system-level and/or Message VPN-level events over the message bus must be enabled.

Enabling publication of subscription-level events to the message bus in a Message VPN may affect event broker subscription performance. This should be turned on only if there are clients that require these messages to be published.

Publishing System Level Events

By default, publication of system-level events over the message bus is not enabled. To enable an event broker to publish system level events, a currently enabled Message VPN must be configured as the Management Message VPN. Only one Message VPN can be configured as the Management Message VPN on an event broker. Once a Management Message VPN is configured, system events are then always published to the Message VPN designated as the Management Message VPN. If no Management Message VPN is configured, then system events are not published for the event broker.

Clients connected to a Management Message VPN receive system level events, and are able to access statistics from all VPNs. By default, management privileges are not assigned to any VPN, and they need to be explicitly configured.

For more information on how to configure a VPN as a Management Message VPN, and enable publishing of system level events, refer to Designating Management Message VPNs.

Publishing Message VPN Level Events

By default, publication of Message VPN client and subscription events over the message bus is not enabled, and must be separately enabled for each Message VPN. Once enabled, management applications using Solace APIs can subscribe to the well-known topics to receive event messages from the Message VPN.

Another option for event monitoring is to integrate your application with a 3rd party monitoring system - open-source or proprietary - that can leverage one or more of the supported integration options.

For information on how to configure publication of Message VPN level events, refer to Configuring Message VPN Event Generation and Configuring Events and Thresholds.

  • For examples of event monitoring over the message bus, refer to sample applications EventMonitor.java or EventMonitor.c, available in the Solace Java and C SDKs. The examples demonstrate how to configure a client to receive event broker event information using Solace APIs.
  • Refer to Subscribing to Message Bus Events for more details on how to configure and subscribe to events over the message bus.

These are examples of Message VPN level events that a monitoring application may subscribe to:

  • VPN_AD_BIND_COUNT_HIGH
  • VPN_AD_MSG_SPOOL_HIGH
  • VPN_AD_MSG_SPOOL_QUOTA_EXCEED
  • VPN_VPN_CONNECTIONS_HIGH
  • VPN_VPN_EG_MSG_RATE_HIGH
  • VPN_VPN_ING_MSG_RATE_HIGH
  • VPN_VPN_SUBSCRIPTIONS_HIGH

For a detailed listing of the possible events, and descriptions of each, refer to the Syslog Event Reference.

Request / Reply Monitoring Using SEMP

SEMP is a request/reply protocol that uses an XML schema to identify all managed objects available in an event broker. Any object available through the Solace CLI or SolAdmin is also available through SEMP.

Applications can use the SEMP interface to manage and monitor an event broker. SEMP requests can be made either through the HTTP port on the administrative interface, or through the messaging port on the messaging interface. The requesting client can send XML-encoded requests to the event broker, and receive corresponding XML-encoded replies. These requests and replies parallel the commands and replies offered through the CLI.

  • The event broker supports a maximum SEMP polling rate of 10 requests per second across all SEMP connections to the event broker. Monitoring applications should be configured so as to not exceed this threshold.
  • If monitoring applications are using SEMP to monitor PubSub+ Cache instances, then it is highly recommended that SEMP requests be kept to a minimum to avoid affecting the performance of PubSub+ Cache while processing live data traffic. The interval at which SEMP requests are sent periodically to monitor PubSub+ Cache should not be less than 30 seconds. This interval should be adjusted based on the peak performance of PubSub+ Cache instances.
  • Refer to SEMP for more details on the SEMP format, and how to send and receive SEMP request/replies.
  • For examples of making SEMP requests over the message bus, refer to the sample application SempGetOverMB.java or sempGetOverMb.c, available in the Java and C SDKs. The examples demonstrate how a SEMP request/response can be accomplished over the message bus.

For more information on the common statistics that can be queried via SEMP on the health of the event broker, refer to Gathering Statistics with SEMP.

Solace Syslog

The event broker generates Syslog messages for events related to system health, system operation, and resource utilization. Syslog messages are logged locally on the event broker, but can also be forwarded to a remote server. For more information, refer to Monitoring Using Syslog.