Monitoring Events Using Syslog

You can use the syslog logging mechanism to monitor Solace PubSub+ event broker events.

Solace PubSub+ software generates syslog messages, as defined in RFC 3164, to record events that occur on the event broker, such as:

  • Routine operations: such as clients connecting and disconnecting, change of command modes
  • Failure and error conditions: such as redundant power supply failures
  • Emergency or critical conditions: such as a blade taken offline due to excessive temperature

Each syslog message identifies the software process or event broker subsystem that generated the message, and briefly describes the operation or error that occurred.

As shown in the figure Syslog Message Flow below, the syslog system offers both the logging of syslog event messages to local files on event brokers, as well as the forwarding of syslog event messages over the network through standard protocols to remote hosts.

A description of syslog messages related to event broker events can be found at Syslog Event Reference.

Syslog Message Format and Components

A syslog message ASCII string consists of:

  • a message priority (facility and severity)
  • a header (timestamp and host)
  • the message string

Each syslog message belongs to a facility, which is a group of messages that are either generated by the same software process, or concern a similar event broker subsystem condition or activity (such as debugging attempts).

Four configurable facility destinations are defined and reserved on the event broker for grouping together related messages from a facility log file: command, debug, event, and system (refer to the table Syslog Facilities below). Syslog message files from the four non‑configurable destinations are all directed to a file on the local syslog message host.

In addition, three configurable facility destinations are available for configuration by users for grouping together related messages from a facility to a log file for forwarding to a remote syslog host.

Syslog Message Flow

Syslog Facilities
Local Syslog Facility Associated Log Type of Event or Error

local0

debug

Actions performed or errors encountered by debug processes.

local1

command

Actions performed or errors encountered by commands issued at the event broker Command Line Interface (CLI) prompt or by CLI script files, or by SEMP.

local3

event

Actions performed or errors encountered by event broker processes, for example, event broker events related to publishers and subscribers, event broker events related to physical links/LAGs, event broker events related to routing protocols, event broker events related to physical hardware.

local4

system

Actions performed or errors encountered by event broker processes that relate to the health of the event broker.

The system log is a subset of the event log, where important logs for monitoring systems are duplicated. The majority of SYSTEM_* events are duplicated, along with many VPN_* events that are noteworthy (e.g. HIGH and _CLEAR events). Note that no CLIENT* events are duplicated.

Each syslog message is also preassigned a severity level, which indicates how seriously the triggering event affects event broker functions. These message event severity levels are defined as an ordered list. The Syslog Message Severity Levels listed below define the message event severity levels, from highest level to lowest.

The associated facility and severity level of a syslog message are together referred to as its priority.

Syslog Message Severity Levels
Severity Level Description

crit

Failures causing many or all services in the system to go down.

err

A service has gone down. An unexpected service outage affecting a small number or single service.

warning

A service has been degraded or is exceeding expected operating thresholds.

notice

Normal but significant conditions; for example, service-affecting configuration changes.

info

Clearing previous raised conditions where no action needs to be taken.

Events to Monitor to Determine the Health of Event Brokers

Solace recommends that an operations team monitor the events contained in the system syslog (system.log or system facility), as this syslog contains all events relevant to the health of the event broker. An operations team may also want to monitor some events in the event.log file depending on the how their messaging network is deployed. For a detailed listing of the possible events and descriptions of each event, refer to Syslog Event Reference.

Configuring Syslog to Forward Messages

You can use a syslog facility destination on an event broker to group together related messages from a facility to a log file so that it can be forwarded to a remote syslog message host. A maximum of three user-configured syslog destinations are permitted per event broker.

Syslog message files from a user-configured destination are not received by remote syslog message hosts (that is, clients) until hosts are designated and configured. You can configure up to three syslog message hosts for each user-configured syslog destination.

  • To create a new syslog facility destination, enter the following command: 
    solace(configure)# create syslog <name>

    • To edit the properties of an existing syslog facility destination, enter the following command:

      solace(configure)# syslog <name>

      Where:

      <name> is the name of the user-configured syslog destination. The syslog name can contain up to 31 alphanumeric characters and must be unique among all created syslog destinations.

      The no version of this command (no syslog) removes the given from the event broker.

For a syslog destination, you can configure the following parameters:

Facility

To add the command, event, or system syslog facility to a user-configured syslog destination, enter the following commands:

solace(configure)# create syslog <name>
solace(configure/syslog)# facility {command | event | system}

Where:

command asks to add the command syslog facility to the user-configured destination

event asks to add the event syslog facility to the user-configured destination

system asks to add the system syslog facility to the user-configured destination

The no version deletes the specified facility from the user-configured destination.

Host

To designate a remote syslog message host as a recipient for syslog files, enter the following commands:

solace(configure)# create syslog <name>
solace(configure/syslog)# host <hostname-or-address> [transport {tcp | udp | tls}]

Where:

<hostname-or-address> is either the name of the remote host, or the IP address with optional port number, specified in the dotted decimal notation form nnn.nnn.nnn.nnn[:nnn]. UDP port 514 is used as a default if the port number is not provided.

transport {tcp | udp | tls} sets the transport mode used for forwarding the syslog file to the remote host to either TCP, UDP, or TLS, respectively. UDP is used as a default if this parameter is not provided. If the transport mode is set to TLS, the CA certificates to trust, the allowed protocol versions, and the cipher suites to accept across the remote rsyslogd TLS connection are specified using the global SSL CLI commands. For more information, see TLS/SSL Encryption.

The no version of this command (no host) removes the specified host from the user-configured destination.

This CLI command sequence example configures the syslog facility destination named bestlogspotever, adds the command and event facilities to this user-configured syslog destination, and designates and configures two syslog message hosts for receipt of the forwarded syslog message file.

solace> enable
solace# configure terminal
solace(configure)# create syslog bestlogspotever
solace(configure/syslog)# facility command
solace(configure/syslog)# facility event
solace(configure/syslog)# host 192.168.1.12 transport tcp
solace(configure/syslog)# host 192.168.1.13 transport tcp

To remove the syslog destination named bestlogspotever once configured, enter:
solace(configure)# no syslog bestlogspotever

Configuring the Message Format for Syslog Facilities

To allow you to more easily understand the contents of log messages, the event broker allows event and system log messages to be emitted from the broker as JSON, with the message arguments in independent JSON properties.

To configure the message format for the event and system syslog facilities, enter the following commands:

solace(configure)# logging
solace(configure/logging)# facility
solace(configure/logging/facility)# {event | system} message-format {text | json}

Where:

event specifies to configure the event syslog facility message format.

system specifies to configure the system syslog facility message format.

text specifies to log message as unstructured text. This is the default value.

json specifies to log message as JSON.

The no form of this command no {event | system} message-format, returns the value to the default.

Log files are not rotated when the format is updated, so at the time of the switch the log file will contain a mix of formats.

JSON Format

When JSON format is enabled, the log message is formatted as JSON (all on a single line) and that JSON is passed to syslog(). The arguments to the log message in Solace PubSub+ Syslog Events is the names of the properties. In addition, the following metadata properties are added at the start of the object in this order:

time — the RFC 3339 timestamp.

solFacility — the Solace name for the syslog facility. Either event or system.

severity — the syslog severity.

host — the hostname.

tag — the event tag.

scope — either VPN, CLIENT, or SYSTEM.

event — the event name, for example CLIENT_CLIENT_DISCONNECT.

vpn — the name of the VPN. Only added if the scope is VPN or CLIENT.

client — the name of the client. Only added if the scope is CLIENT.

raise — an identifier to allow correlation of raise and clear events; also serves to indicate that this event is a raise event. Only added for raise events.

clear — an identifier to allow correlation of raise and clear events; also serves to indicate that this event is a clear event. Only added for clear events.

After the metadata, the arguments follow in the order they are defined. A msg property is added at the end with the event text (with substitutions).

Setting the Maximum Size of JSON Format Syslog Messages

You can set the maximum size of JSON format syslog messages. If the full JSON would exceed that size, properties are dropped or truncated from the end of the message until the value fits within the specified size with an additional property logIncomplete added with the value 1. Only the msg property is truncated, all other properties are dropped if the full value doesn't fit.

Setting a maximum size is useful because the event broker will remove properties or truncate the value of the msg property, but will still produce valid JSON. Downstream processes will generally just truncate the log line, resulting in invalid JSON.

To set the maximum size of JSON format syslog messages, enter the following commands:

solace(configure)# logging
solace(configure/logging)# max-json-message-size <max-size>

Where:

<max-size> is the maximum size in bytes of JSON format syslog messages. If a remote syslog destination is configured, the syslog header is included. The valid range of values is 1024 to 8192. The default value is 8192.

The no version of this command, no max-json-message-size, returns the value to the default.

Viewing Syslog Status

To view the configuration of syslog on your event broker, enter the following command:

solace> show syslog [<name>]

Where:

<name> is the name of the syslog facility destination. Entering no name displays all destinations. 

solace> show syslog bestlogspotever
------------------------------------------------
Name: bestlogspotever
Facilities: event command
Files
Hosts                            Transport
  192.168.1.12                   TCP
  192.168.1.13                   TCP
solace> show syslog
------------------------------------------------
Name: command
Facilities: command
Files
  command.log
Hosts                            Transport
------------------------------------------------
Name: debug
Facilities: debug
Files
  debug.log
Hosts                            Transport
------------------------------------------------
Name: event
Facilities: event
Files
  event.log
Hosts                            Transport
------------------------------------------------
Name: system
Facilities: system
Files
  system.log
Hosts                            Transport
------------------------------------------------
Name: scott
Facilities: event command
Files
Hosts                            Transport
  192.168.1.12                   TCP
  192.168.1.13                   TCP

Managing Local Syslog Files

Syslog record files are uncompressed text files that have a <filename>.log.[x] format, where <filename> can be named after a command, event, system, or debug, and x is an integer from 1 to 20 that identifies the archived file number. The active file is named <filename>.log with no numeric suffix.

Retrieving Syslog Record Files

Syslog record files are stored locally as uncompressed text files in the /logs subdirectory on the event broker. To retrieve and view the command log record files, use these CLI commands :

  • copy — transfers a text file from the /logs subdirectory to an external SFTP or SCP server from the event broker.
    Example:
    solace# copy /logs/command.log sftp://john@10.11.12.13/home/john/solacelogs/
  • more — directly displays the contents of a text file from the /logs subdirectory on the event broker.
    Example:
    solace> more /logs/system.log

Logging Capacity

The active syslog record file closes once the file size exceeds 50 MB. Upon closing, a new file opens, and this cycle repeats. The syslog record text files are in the /logs directory and can be retrieved from event brokers. You can view up to 21 syslog record text files for every local syslog facility at any given time, including the active log file. To stay within the 21 file limit, when a new log file is created, the oldest log file is discarded to make space for the new log file. Retention of log files is subject to available disk space. The PubSub+ software event broker caps the space allocated to logs at 1GB.

Syslog record files may contain personal information. To learn more about how to protect this data, see Configuring the Log File Retention Policy.

Directory Maintenance

Because syslog record files are rotated, directory maintenance is not required.

Managing Command Logging

The command logging facility is used to capture information about all event broker commands issued by users through either the Solace CLI or the SEMP interface.

Command Log Record Data Example:

2016-05-02T10:24:51.162-05:00 <local1.info> solace1 admin[12694]: CLI/2        192.168.1.217    admin             10:24:47  10:24:48  ok                              (configure/message-vpn)# max-connections 2000

The information contained in each command log record includes, in order:

  • log date and time, for example, 2016-05-02T10:24:51.162-05:00
  • syslog destination and level, for example, <local1.info>
  • hostname of the event broker, for example, solace1
  • local user name with local process identifier number or pid in square brackets, for example, admin[12694]
  • type of event broker interface making the command attempt, for example, CLI or SEMP: CLI/2
  • IP address of host of source of command attempt, for example, 192.168.1.217
  • user name assigned to the CLI or SEMP user account
  • time when command attempt was started, for example, 10:24:47
  • time when command attempt finished, for example, 10:24:48
  • description of result, that is: ok, command action status, or reason for failure
  • command operation attempted: (configure/message-vpn)# max-connections 2000

Configuring Command Logging

To configure the command logging feature on an event broker, which logs user actions and commands to a viewable command log file, enter the following CLI commands:

solace(configure)# logging
solace(configure/logging)# command {cli | semp-mgmt | semp-msgbus| all} mode {shutdown | config-cmds | all-cmds}

Where:

cli, semp-mgmt, semp-msgbus, or all specify the event broker interface to be logged

shutdown turns off the command logging facility for the specified event broker interface

config-cmds specifies log only configuration commands (not show commands) for the specified event broker interface

all-cmds specifies log all commands except for help (help commands are never logged) for the specified event broker interface

The no version of this command (no command {cli | semp | all}) reverts command logging back to the default mode of logging only configuration commands for the specified event broker interface (that is, cli, semp-mgmt, semp-msgbus, or all).

Viewing Command Logging Configuration

To view the configuration of the command logging facility on the event broker, enter the following CLI command:

solace> show logging command
solace> show logging command
 
Cmd Interface       Logging Mode
--------------------------------
CLI                 config
SEMP/mgmt           config
SEMP/msgbus         config

Interpreting Client Disconnect Event Logs

Whenever clients are disconnected from the event broker, the event broker reports the reason for the disconnect through a disconnect event log. This log reports the final statistic summary event for the connection.

SMF Client Disconnect Event Log Example

2013-05-06T18:18:56-0400 <local3.info> solace1 event: CLIENT: CLIENT_CLIENT_DISCONNECT: testVpn testHost/17879/#00010001 Client (82) testHost/17879/#00010001 username testUser WebSessionId (N/A) reason(Peer TCP Closed) final statistics - dp(103, 103, 0, 0, 103, 103, 1557, 2373, 0, 0, 1557, 2373, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) conn(0, 0, 192.168.164.34:58214, CLSWT, 0, 0, 0) zip(0, 0, 0, 0, 0.00, 0.00, 0, 0, 0, 0, 0, 0, 0, 0) web(0, 0, 0, 0, 0, 0, 0), SslVersion(), SslCipher()

MQTT Client Disconnect Event Log Example

2017-07-03T15:59:23+0800 <local3.info> sgdemo1 event: CLIENT: CLIENT_CLIENT_DISCONNECT_MQTT: plant_monitor #mqtt/PLANT_MONITOR_192.168.42.250/71841 Client (1141) #mqtt/PLANT_MONITOR_192.168.42.250/71841 username default ClientId (PLANT_MONITOR_192.168.42.250) reason(Client Disconnect Received) final statistics - dp(2, 1, 1, 0, 3, 1, 80, 4, 238, 0, 318, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) conn(0, 0, 192.168.42.1:20362, ESTAB, 0, 0, 0) mqtt(1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), SslVersion(), SslCipher(), WillSent(0)

The reason entry describes why the client was disconnected:

  • (Client Disconnect Received)—Connection dropped because a disconnect notification was received.
  • (Document Too Large)—Connection dropped because an incoming message is too large.
  • (Forced Logout)—Administrative disconnect or shutdown action triggered either by a client disconnect, or by replacement of duplicate client connections during authentication.

    • Replacement of duplicate client connections during authentication is by default enabled on the event broker. To disable, enter the following command: solace(configure)# no authentication replace-duplicate-client-connections.

  • (Inactive Connection)—Connection dropped due to an inactive network connection.
  • (Peer Draining Messages Too Slowly)—Connection dropped due to low throughput while the event broker is under stress. During stressful conditions, the event broker drops connections that are moving too slowly to free up system resources.
  • (Peer Host Unreachable)—The event broker is unable to reach the remote host.
  • (Peer Refused TCP Connection)—For a connection initiated by the event broker (for example, Message VPN bridging and event broker-to-event broker connections), the peer refused the TCP connection.
  • (Peer TCP Closed)—The remote end has gracefully closed the TCP connection.
  • (Peer TCP Reset)—The remote end has ungracefully closed the TCP connection.
  • (SSL CTX has been expired)—The oldest TLS context (data derived from the server certificate) in the event broker expired. Clients using that TLS context are disconnected.
  • (TCP Keep-Alive Failure)—Connection dropped due to a failure on the remote end to respond to a TCP Keepalive probe.
  • (Too Many TCP Retransmissions)—Connection dropped due to bad throughput.

The dp entry lists the following client dataplane statistics, in order:

  • Control Messages Received
  • Control Message Delivered
  • Topic Messages Received
  • Topic Messages Delivered
  • Total Messages Received
  • Total Message Delivered
  • Control Bytes Received
  • Control Bytes Delivered
  • Topic Bytes Received
  • Topic Bytes Delivered
  • Total Bytes Received
  • Total Bytes Delivered
  • Current Message Rate at Ingress
  • Current Message Rate at Egress
  • Average Message Rate at Ingress
  • Average Message Rate at Egress
  • Denied Duplicate Clients

(Messages Discarded at Ingress due to:)

  • No Subscription Match
  • Topic Parse Error
  • Internal Parse Error
  • Message Too Big

(Queue Discards due to:)

  • Transmit Congestion

The conn entry lists the following client connection statistics information, in order:

  • Received Queue Bytes
  • Send Queue Bytes
  • Client Address: Port
  • State
  • Received Out of Order
  • Fast Retransmit
  • Timed Retransmit

The zip entry lists the following client compression statistics information, in order:

  • Compressed Bytes Received
  • Compressed Bytes Delivered
  • Uncompressed Bytes Received
  • Uncompressed Bytes Delivered
  • Compression Ratio at Ingress
  • Compression Ratio at Egress
  • Current Compressed Byte Rate at Ingress
  • Current Compressed Byte Rate at Egress
  • Current Uncompressed Byte Rate at Ingress
  • Current Uncompressed Byte Rate at Egress
  • Average Compressed Byte Rate at Ingress
  • Average Compressed Byte Rate at Egress
  • Average Uncompressed Byte Rate at Ingress
  • Average Uncompressed Byte Rate at Egress

The web entry lists the following Web client dataplane statistics information, in order:

  • Web Messages Received—total number of HTTP requests received by the event broker for the client session.
  • Web Messages Delivered—total number of HTTP responses sent by the event broker for the client session.
  • Web Bytes Received—total number of octets in the message bodies of all HTTP requests received by the event broker for the client session.
  • Web Bytes Delivered—total number of octets in the message bodies of all HTTP responses sent by the event broker for the client session.
  • Web Out Of Order—approximate number of TCP out-of-order reception events detected by the event broker, which occur while the TCP connection is in use by the client session.
  • Web Fast Retransmit—approximate number of TCP fast retransmission events initiated by the event broker after receiving duplicate TCP acknowledgments, which occur while the TCP connection is in use by the client session.
  • Web Timed Retransmit—approximate number of TCP timed retransmission events initiated by the event broker after not receiving TCP acknowledgments, which occur while the TCP connection is in use by the client session.

The mqtt entry lists the following control packet statistics for MQTT clients, in order:

  • CONNECT packets received
  • CONNACK packets sent
  • CONNACK Errors sent
  • PUBREC—Publish Received responses
  • PUBLISH—Publish Message packets sent
  • PUBACK—Publish Acknowledgements Received
  • PUBACK—Publish Acknowledgements sent
  • PUBREC—Publish Received packets sent
  • PUBREL—Publish Release responses received
  • PUBCOMP—Publish Complete responses sent
  • SUBSCRIBE—Subscribe to Topics packets received
  • SUBACK—Subscribe Acknowledgements sent
  • SUBACK—Subscribe Acknowledgement errors sent
  • UNSUBSCRIBE—Unsubscribe from Topic packets received
  • UNSUBSCRIBE—Unsubscribe from Topic packets sent
  • PINGREQ—PING Requests received
  • PINGRESP—PING Responses sent
  • DISCONNECT—Disconnect notifications received

    For information on the various MQTT control packets, see 3 MQTT Control Packets in the MQTT Version 3.1.1 Protocol Conformance document.

Where SslVersion lists the TLS/SSL protocol version used.

Where SslCipher lists the cipher suite used.

Interpreting Flow Event Logs

Whenever a flow is closed or unbound from an endpoint on the event broker, the event broker reports the reason for it through a flow event log. This log reports the final statistic summary event for the flow close or unbind, as applicable.

Flow Close Event Log Example

2015-01-08T09:05:40+0000 <local3.info> lab-128-68 event: CLIENT: CLIENT_CLIENT_CLOSE_FLOW: solace dev214/9997/#00010001 Client (6119) dev214/9997/#00010001 username default Pub flow session flow name 99f5f4aa343a4e77bc45f6b3c4db2946 (8048), publisher id 5949, last message id 565, window size 50, final statistics - flow(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 565)

The flow entry lists the following flow close statistics, in order:

  • Spooling Not Ready
  • Out of Order Messages
  • Duplicate Messages
  • No Eligible Destinations
  • Spool Over Quota
  • Queue/Topic-Endpoint Over
  • Replay Log Over Quota
  • Maximum Message Usage Exceeded
  • Maximum Message Size Exceeded
  • Remote Router Spooling Not Supported
  • Spool To ADB Fail
  • Spool To Disk Fail
  • Errored Message
  • Queue Not Found
  • Spool Shutdown Discard
  • User Profile Deny Guaranteed
  • No Local Delivery Discard
  • SMF TTL Exceeded
  • Publish ACL Denied
  • Destination Group Error
  • Not Compatible With Forwarding Mode
  • Low-Priority-Msg Congestion Discard
  • Spool File Limit Exceeded
  • Replication Is Standby Discard
  • Sync Replication Ineligible Discard
  • Messages Received

    • Descriptions for these flow close statistics can be found in Description of Detailed Spooled Message Statistics.

Flow Unbind Event Log Example

CLIENT: CLIENT_CLIENT_UNBIND: RP dev3-185/30296/00030001/349CSHv5cC Client (66) dev3-185/30296/00030001/349CSHv5cC username solclient_1 Unbind to Flow Id (11), ForwardingMode(StoreAndForward), final statistics - flow(255, 0, 0, 0, 0, 2, 0, 0, 2, 0), isActive(No), Reason(Client issued unbind)

The flow entry lists the following flow unbind statistics, in order:

  • Message Spool Window Size
  • Used Window
  • Unacked Messages
  • Low Message ID AckPending
  • High Message ID AckPending
  • Window Closed
  • Message Redelivered
  • Message Transport Retransmits
  • Message Confirmed Delivered Store and Forward
  • Message Delivered Cut Through