Monitoring Events Using Syslog

You can use the syslog logging mechanism to monitor Solace PubSub+ message broker events.

Solace PubSub+ software generates syslog messages, as defined in RFC 3164, to record events that occur on the message broker, such as:

  • routine operations, such as creation and deletion of CSPF neighbors, change of command modes
  • failure and error conditions, such as redundant power supply failures
  • emergency or critical conditions, such as a blade taken offline due to excessive temperature

Each syslog message identifies the software process or message broker subsystem that generated the message, and briefly describes the operation or error that occurred.

As shown in the figure Syslog Message Flow below, the syslog system offers both the logging of syslog event messages to local files on message brokers, as well as the forwarding of syslog event messages over the network through standard protocols to remote hosts.

A description of syslog messages related to message broker events can be found at Solace PubSub+ Event Reference.

Syslog Message Format and Components

A syslog message ASCII string consists of:

  • a message priority (facility and severity)
  • a header (timestamp and host)
  • the message string

Each syslog message belongs to a facility, which is a group of messages that are either generated by the same software process, or concern a similar message broker subsystem condition or activity (such as debugging attempts).

Four configurable facility destinations are defined and reserved on the message broker for grouping together relates messages from a facility log file: command, debug, event, and system (refer to the table Syslog Facilities below). Syslog message files from the four non‑configurable destinations are all directed to a file on the local syslog message host.

In addition, three configurable facility destinations are available for configuration by users for grouping together related messages from a facility to a log file for forwarding to a remote syslog host.

Syslog Message Flow

Syslog Facilities

Local Syslog Facility

Associated Log

Type of Event or Error

local0

debug

Actions performed or errors encountered by debug processes.

local1

command

Actions performed or errors encountered by commands issued at the message broker Command Line Interface (CLI) prompt or by CLI script files, or by SEMP.

local3

event

Actions performed or errors encountered by message broker processes, for example, message broker events related to publishers and subscribers, message broker events related to physical links/LAGs, message broker events related to routing protocols, message broker events related to physical hardware.

local4

system

Actions performed or errors encountered by message broker processes that relate to the health of the message broker.

The system log is a subset of the event log, where important logs for monitoring systems are duplicated. The majority of SYSTEM_* events are duplicated, along with many VPN_* events that are noteworthy (e.g. HIGH and _CLEAR events). Note that no CLIENT* events are duplicated.

Each syslog message is also preassigned a severity level, which indicates how seriously the triggering event affects message broker functions. These message event severity levels are defined as an ordered list. The Syslog Message Severity Levels listed below define the message event severity levels, from highest level to lowest.

The associated facility and severity level of a syslog message are together referred to as its priority.

Syslog Message Severity Levels

Severity Level

Description

crit

Failures causing many or all services in the system to go down.

err

A service has gone down. An unexpected service outage affecting a small number or single service.

warning

A service has been degraded or is exceeding expected operating thresholds.

notice

Normal but significant conditions; for example, service-affecting configuration changes.

info

Clearing previous raised conditions where no action needs to be taken.

Events to Monitor to Determine the Health of Message Brokers

Solace recommends that an operations team monitor the events contained in the system syslog (system.log or system facility), as this syslog contains all events relevant to the health of the message broker. An operations team may also want to monitor some events in the event.log file depending on the how their messaging network is deployed. For a detailed listing of the possible events and descriptions of each event, refer to Solace PubSub+ Event Reference.

Configuring Syslog to Forward Messages

You can use a syslog facility destination on a message broker to group together related messages from a facility to a log file so that it can be forwarded to a remote syslog message host. A maximum of three user-configured syslog destinations are permitted per message broker.

Syslog message files from a user-configured destination are not received by remote syslog message hosts (that is, clients) until hosts are designated and configured. You can configure up to three syslog message hosts for each user-configured syslog destination.

  • To create a new syslog facility destination, enter the following command:

    solace(configure)# create syslog <name>

  • To edit the properties of an existing syslog facility destination, enter the following command:

    solace(configure)# syslog <name>

    Where:

    <name> is the name of the user-configured syslog destination. The syslog name can contain up to 31 alphanumeric characters and must be unique among all created syslog destinations.

    Note:  The no version of this command (no syslog) removes the given syslog destination from the message broker.

For a syslog destination, you can configure the following parameters:

Facility

To add the command or event syslog facility to a user-configured syslog destination, enter the following commands:

solace(configure)# create syslog <name>
solace(configure/syslog)# facility {command | event | system}

Where:

command asks to add the command syslog facility to the user-configured destination

event asks to add the event syslog facility to the user-configured destination

system asks to add the system syslog facility to the user-configured destination

Note:  The no version deletes the specified facility from the user-configured destination.

Host

To designate a remote syslog message host as a recipient for syslog files, enter the following commands:

solace(configure)# create syslog <name>
solace(configure/syslog)# host <hostname-or-address> [transport {tcp | udp}]

Where:

<hostname-or-address> is either the name of the remote host, or the IP address with optional port number, specified in the dotted decimal notation form nnn.nnn.nnn.nnn[:nnn]. UDP port 514 is used as a default if the port number is not provided.

transport {tcp | udp} sets the transport mode used for forwarding the syslog file to the remote host to either TCP or UDP, respectively. UDP is used as a default if this parameter is not provided.

Note:  The no version of this command (no host) removes the specified host from the user-configured destination.

Viewing Syslog Status

To view the configuration of syslog on your message broker, enter the following command:

solace> show syslog [<name>]

Where:

<name> is the name of the syslog facility destination. Entering no name displays all destinations.

Managing Local Syslog Files

Syslog record files are uncompressed text files with the name <filename>.log.[x], where <filename> is one of command, event, system or debug and x is an integer from 1 to 20 identifying the archived file number. The active file is named <filename>.log. with no numeric suffix.

Retrieving Syslog Record Files

Syslog record files are stored locally on the message broker in subdirectory /logs as uncompressed text files. The following CLI commands can be used to retrieve and view the command log record files:

  • copy— transfers a text file from the /logs subdirectory to an external SFTP or SCP server from the message broker for viewing.
    Example:

    solace# copy /logs/command.log sftp://john@10.11.12.13/home/john/solacelogs/

  • more —directly displays the contents of a text file from the /logs subdirectory on the message broker for viewing.
    Example:

    solace> more /logs/system.log

Logging Capacity

The active syslog record file is closed once the file size exceeds 50 MB. Upon closing, a new file is opened, and this cycle repeats. The syslog record text files are in the /logs directory and can be retrieved from message brokers. A total of up to 21 syslog record text files for every local syslog facility are available for viewing at any one time, including the currently active log file. To keep within the 21 file limit, when a new log file is created, the oldest log file is discarded to make space for the new log file.

Directory Maintenance

Directory maintenance is not required for syslog record files as the files are rotated.

Managing Command Logging

The command logging facility is used to capture information about all message broker commands issued by users through either the Solace CLI or the SEMP interface.

Command Log Record Data Example:

2016-05-02T10:24:51.162-05:00 <local1.info> solace1 admin[12694]: CLI/2 192.168.1.217 admin 10:24:47 10:24:48 ok (configure/message-vpn)# max-connections 2000

The information contained in each command log record includes, in order:

  • log date and time, for example, 2016-05-02T10:24:51.162-05:00
  • syslog destination and level, for example, <local1.info>
  • hostname of the message broker, for example, solace1
  • local user name with local process identifier number or pid in square brackets, for example, admin[12694]
  • type of message broker interface making the command attempt, for example, CLI or SEMP: CLI/2
  • IP address of host of source of command attempt, for example, 192.168.1.217
  • user name assigned to the CLI or SEMP user account
  • time when command attempt was started, for example, 10:24:47
  • time when command attempt finished, for example, 10:24:48
  • description of result, that is: ok, command action status, or reason for failure
  • command operation attempted: (configure/message-vpn)# max-connections 2000

Configuring Command Logging

To configure the command logging feature on a message broker, which logs user actions and commands to a viewable command log file, enter the following CLI commands:

solace(configure)# logging
solace(configure/logging)# command {cli | semp-mgmt | semp-msgbus| all} mode {shutdown | config-cmds | all-cmds}

Where:

cli, semp-mgmt, semp-msgbus, or all specify the message broker interface to be logged

shutdown turns off the command logging facility for the specified message broker interface

config-cmds specifies log only configuration commands (not show commands) for the specified message broker interface

all-cmds specifies log all commands except for help (help commands are never logged) for the specified message broker interface

Note:  The no version of this command (no command {cli | semp | all}) reverts the command logging facility back to the default mode of config for the specified message broker interface (that is, cli, semp-mgmt, semp-msgbus, or all).

Viewing Command Logging Configuration

To view the configuration of the command logging facility on the message broker, enter the following CLI command:

solace> show logging command

Interpreting Client Disconnect Event Logs

Whenever clients are disconnected from the message broker, the message broker reports the reason for the disconnect through a disconnect event log. This log reports the final statistic summary event for the connection.

SMF Client Disconnect Event Log Example

2013-05-06T18:18:56-0400 <local3.info> solace1 event: CLIENT: CLIENT_CLIENT_DISCONNECT: testVpn testHost/17879/#00010001 Client (82) testHost/17879/#00010001 username testUser WebSessionId (N/A) reason(Peer TCP Closed) final statistics - dp(103, 103, 0, 0, 103, 103, 1557, 2373, 0, 0, 1557, 2373, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) conn(0, 0, 192.168.164.34:58214, CLSWT, 0, 0, 0) zip(0, 0, 0, 0, 0.00, 0.00, 0, 0, 0, 0, 0, 0, 0, 0) web(0, 0, 0, 0, 0, 0, 0), SslVersion(), SslCipher()

MQTT Client Disconnect Event Log Example

2017-07-03T15:59:23+0800 <local3.info> sgdemo1 event: CLIENT: CLIENT_CLIENT_DISCONNECT_MQTT: plant_monitor #mqtt/PLANT_MONITOR_192.168.42.250/71841 Client (1141) #mqtt/PLANT_MONITOR_192.168.42.250/71841 username default ClientId (PLANT_MONITOR_192.168.42.250) reason(Client Disconnect Received) final statistics - dp(2, 1, 1, 0, 3, 1, 80, 4, 238, 0, 318, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) conn(0, 0, 192.168.42.1:20362, ESTAB, 0, 0, 0) mqtt(1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), SslVersion(), SslCipher(), WillSent(0)

The reason entry describes why the client was disconnected:

  • (Client Disconnect Received)—Connection dropped because a disconnect notification was received.
  • (Forced Logout)—Administrative disconnect or shutdown action triggered either by a client disconnect, or by replacement of duplicate client connections during authentication.
  • Note:  Replacement of duplicate client connections during authentication is by default enabled on the message broker. To disable, enter the following command: solace(configure)# no authentication replace-duplicate-client-connections.

  • (Inactive Connection)—Connection dropped due to an inactive network connection.
  • (Peer Draining Messages Too Slowly)—Connection dropped due to low throughput while the message broker is under stress. During stressful conditions, the message broker drops connections that are moving too slowly to free up system resources.
  • (Peer Refused TCP Connection)—For a connection initiated by the message broker (for example, Message VPN bridging and message broker-to-message broker connections), the peer refused the TCP connection.
  • (Peer TCP Closed)—The remote end has gracefully closed the TCP connection.
  • (Peer TCP Reset)—The remote end has ungracefully closed the TCP connection.
  • (TCP Keep-Alive Failure)—Connection dropped due to a failure on the remote end to respond to a TCP Keepalive probe.
  • (Too Many TCP Retransmissions)—Connection dropped due to bad throughput.

The dp entry lists the following client dataplane statistics, in order:

  • Control Messages Received
  • Control Message Delivered
  • Topic Messages Received
  • Topic Messages Delivered
  • Total Messages Received
  • Total Message Delivered
  • Control Bytes Received
  • Control Bytes Delivered
  • Topic Bytes Received
  • Topic Bytes Delivered
  • Total Bytes Received
  • Total Bytes Delivered
  • Current Message Rate at Ingress
  • Current Message Rate at Egress
  • Average Message Rate at Ingress
  • Average Message Rate at Egress
  • Denied Duplicate Clients

(Messages Discarded at Ingress due to:)

  • No Subscription Match
  • Topic Parse Error
  • Internal Parse Error
  • Message Too Big

(Queue Discards due to:)

  • Transmit Congestion

The conn entry lists the following client connection statistics information, in order:

  • Received Queue Bytes
  • Send Queue Bytes
  • Client Address: Port
  • State
  • Received Out of Order
  • Fast Retransmit
  • Timed Retransmit

The zip entry lists the following client compression statistics information, in order:

  • Compressed Bytes Received
  • Compressed Bytes Delivered
  • Uncompressed Bytes Received
  • Uncompressed Bytes Delivered
  • Compression Ratio at Ingress
  • Compression Ratio at Egress
  • Current Compressed Byte Rate at Ingress
  • Current Compressed Byte Rate at Egress
  • Current Uncompressed Byte Rate at Ingress
  • Current Uncompressed Byte Rate at Egress
  • Average Compressed Byte Rate at Ingress
  • Average Compressed Byte Rate at Egress
  • Average Uncompressed Byte Rate at Ingress
  • Average Uncompressed Byte Rate at Egress

The web entry lists the following Web client dataplane statistics information, in order:

  • Web Messages Received—total number of HTTP requests received by the message broker for the client session.
  • Web Messages Delivered—total number of HTTP responses sent by the message broker for the client session.
  • Web Bytes Received—total number of octets in the message bodies of all HTTP requests received by the message broker for the client session.
  • Web Bytes Delivered—total number of octets in the message bodies of all HTTP responses sent by the message broker for the client session.
  • Web Out Of Order—approximate number of TCP out-of-order reception events detected by the message broker, which occur while the TCP connection is in use by the client session.
  • Web Fast Retransmit—approximate number of TCP fast retransmission events initiated by the message broker after receiving duplicate TCP acknowledgments, which occur while the TCP connection is in use by the client session.
  • Web Timed Retransmit—approximate number of TCP timed retransmission events initiated by the message broker after not receiving TCP acknowledgments, which occur while the TCP connection is in use by the client session.

The mqtt entry lists the following control packet statistics for MQTT clients, in order:

  • CONNECT packets received
  • CONNACK packets sent
  • CONNACK Errors sent
  • PUBREC—Publish Received responses
  • PUBLISH—Publish Message packets sent
  • PUBACK—Publish Acknowledgements Received
  • PUBACK—Publish Acknowledgements sent
  • PUBREC—Publish Received packets sent
  • PUBREL—Publish Release responses received
  • PUBCOMP—Publish Complete responses sent
  • SUBSCRIBE—Subscribe to Topics packets received
  • SUBACK—Subscribe Acknowledgements sent
  • SUBACK—Subscribe Acknowledgement errors sent
  • UNSUBSCRIBE—Unsubscribe from Topic packets received
  • UNSUBSCRIBE—Unsubscribe from Topic packets sent
  • PINGREQ—PING Requests received
  • PINGRESP—PING Responses sent
  • DISCONNECT—Disconnect notifications received

    Note:  For information on the various MQTT control packets, see 3 MQTT Control Packets in the MQTT Version 3.1.1 Protocol Conformance document.

Where SslVersion lists the TLS/SSL protocol version used.

Where SslCipher lists the cipher suite used.

Interpreting Flow Event Logs

Whenever a flow is closed or unbound from an endpoint on the message broker, the message broker reports the reason for it through a flow event log. This log reports the final statistic summary event for the flow close or unbind, as applicable.

Flow Close Event Log Example

2015-01-08T09:05:40+0000 <local3.info> lab-128-68 event: CLIENT: CLIENT_CLIENT_CLOSE_FLOW: solace dev214/9997/#00010001 Client (6119) dev214/9997/#00010001 username default Pub flow session flow name 99f5f4aa343a4e77bc45f6b3c4db2946 (8048), publisher id 5949, last message id 565, window size 50, final statistics - flow(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 565)

The flow entry lists the following flow close statistics, in order:

  • Spooling Not Ready
  • Out of Order Messages
  • Duplicate Messages
  • No Eligible Destinations
  • Spool Over Quota
  • Queue/Topic-Endpoint Over Quota
  • Maximum Message Usage Exceeded
  • Maximum Message Size Exceeded
  • Remote Router Spooling Not Supported
  • Spool To ADB Fail
  • Spool To Disk Fail
  • Errored Message
  • Queue Not Found
  • Spool Shutdown Discard
  • User Profile Deny Guaranteed
  • No Local Delivery Discard
  • SMF TTL Exceeded
  • Publish ACL Denied
  • Destination Group Error
  • Not Compatible With Forwarding Mode
  • Low-Priority-Msg Congestion Discard
  • Spool File Limit Exceeded
  • Replication Is Standby Discard
  • Sync Replication Ineligible Discard
  • Messages Received
  • Note:  Descriptions for these flow close statistics can be found in Description of Detailed Spooled Message Statistics.

Flow Unbind Event Log Example

2011-08-29T16:32:14+0000 <local3.notice> lab-130-91 event: CLIENT: CLIENT_CLIENT_UNBIND: solace dev213/31753/#00010001 Client (8) dev213/31753/#00010001 username testUser Unbind to Flow Id (3), ForwardingMode(StoreAndForward), final statistics - flow(255, 0, 0, 0, 0, 0, 0, 100, 0)

The flow entry lists the following flow unbind statistics, in order:

  • Message Spool Window Size
  • Used Window
  • Unacked Messages
  • Low Message ID AckPending
  • High Message ID AckPending
  • Window Closed
  • Message Redelivered
  • Message Confirmed Delivered Store and Forward
  • Message Confirmed Delivered Cut Through