Monitoring Events Using Syslog

You can use the syslog logging mechanism to monitor Solace PubSub+ event broker events.

Solace PubSub+ software generates syslog messages, as defined in RFC 3164, to record events that occur on the event broker, such as:

Routine operations: such as clients connecting and disconnecting, change of command modes
Failure and error conditions: such as redundant power supply failures
Emergency or critical conditions: such as a blade taken offline due to excessive temperature

Each syslog message identifies the software process or event broker subsystem that generated the message, and briefly describes the operation or error that occurred.

As shown in the figure Syslog Message Flow below, the syslog system offers both the logging of syslog event messages to local files on event brokers, as well as the forwarding of syslog event messages over the network through standard protocols to remote hosts.

A description of syslog messages related to event broker events can be found at Syslog Event Reference.

Syslog Message Format and Components

A syslog message ASCII string consists of:

a message priority (facility and severity)
a header (timestamp and host)
the message string

Each syslog message belongs to a facility, which is a group of messages that are either generated by the same software process, or concern a similar event broker subsystem condition or activity (such as debugging attempts).

Four configurable facility destinations are defined and reserved on the event broker for grouping together related messages from a facility log file: command, debug, event, and system (refer to the table Syslog Facilities below). Syslog message files from the four non‑configurable destinations are all directed to a file on the local syslog message host.

In addition, three configurable facility destinations are available for configuration by users for grouping together related messages from a facility to a log file for forwarding to a remote syslog host.

Syslog Message Flow

Syslog Facilities
Local Syslog Facility	Associated Log	Type of Event or Error
local0	debug	Actions performed or errors encountered by debug processes.
local1	command	Actions performed or errors encountered by commands issued at the event broker Command Line Interface (CLI) prompt or by CLI script files, or by SEMP.
local3	event	Actions performed or errors encountered by event broker processes, for example, event broker events related to publishers and subscribers, event broker events related to physical links/LAGs, event broker events related to routing protocols, event broker events related to physical hardware.
local4	system	Actions performed or errors encountered by event broker processes that relate to the health of the event broker. The system log is a subset of the event log, where important logs for monitoring systems are duplicated. The majority of SYSTEM_* events are duplicated, along with many VPN_* events that are noteworthy (e.g. HIGH and _CLEAR events). Note that no CLIENT* events are duplicated.

Each syslog message is also preassigned a severity level, which indicates how seriously the triggering event affects event broker functions. These message event severity levels are defined as an ordered list. The Syslog Message Severity Levels listed below define the message event severity levels, from highest level to lowest.

The associated facility and severity level of a syslog message are together referred to as its priority.

Syslog Message Severity Levels
Severity Level	Description
crit	Failures causing many or all services in the system to go down.
err	A service has gone down. An unexpected service outage affecting a small number or single service.
warning	A service has been degraded or is exceeding expected operating thresholds.
notice	Normal but significant conditions; for example, service-affecting configuration changes.
info	Clearing previous raised conditions where no action needs to be taken.

Events to Monitor to Determine the Health of Event Brokers

Solace recommends that an operations team monitor the events contained in the system syslog (system.log or system facility), as this syslog contains all events relevant to the health of the event broker. An operations team may also want to monitor some events in the event.log file depending on the how their messaging network is deployed. For a detailed listing of the possible events and descriptions of each event, refer to Syslog Event Reference.

Configuring Syslog to Forward Messages

You can use a syslog facility destination on an event broker to group together related messages from a facility to a log file so that it can be forwarded to a remote syslog message host. A maximum of three user-configured syslog destinations are permitted per event broker.

Syslog message files from a user-configured destination are not received by remote syslog message hosts (that is, clients) until hosts are designated and configured. You can configure up to three syslog message hosts for each user-configured syslog destination.

To create a new syslog facility destination, enter the following command:
```
solace(configure)# create syslog <name>
```
To edit the properties of an existing syslog facility destination, enter the following command:
```
solace(configure)# syslog <name>
```
Where:
<name> is the name of the user-configured syslog destination. The syslog name can contain up to 31 alphanumeric characters and must be unique among all created syslog destinations.
The no version of this command (no syslog) removes the given syslog destination from the event broker.

For a syslog destination, you can configure the following parameters:

Facility
Host

Facility

To add the command, event, or system syslog facility to a user-configured syslog destination, enter the following commands:

solace(configure)# create syslog <name>
solace(configure/syslog)# facility {command | event | system}

Where:

command asks to add the command syslog facility to the user-configured destination

event asks to add the event syslog facility to the user-configured destination

system asks to add the system syslog facility to the user-configured destination

The no version deletes the specified facility from the user-configured destination.

Host

To designate a remote syslog message host as a recipient for syslog files, enter the following commands:

solace(configure)# create syslog <name>
solace(configure/syslog)# host <hostname-or-address> [transport {tcp | udp | tls}]

Where:

<hostname-or-address> is either the name of the remote host, or the IP address with optional port number, specified in the dotted decimal notation form nnn.nnn.nnn.nnn[:nnn]. UDP port 514 is used as a default if the port number is not provided.

transport {tcp | udp | tls} sets the transport mode used for forwarding the syslog file to the remote host to either TCP, UDP, or TLS, respectively. UDP is used as a default if this parameter is not provided. If the transport mode is set to TLS, the CA certificates to trust, the allowed protocol versions, and the cipher suites to accept across the remote rsyslogd TLS connection are specified using the global SSL CLI commands. For more information, see TLS/SSL Encryption.

The no version of this command (no host) removes the specified host from the user-configured destination.

Configuring the Message Format for Syslog Facilities

To allow you to more easily understand the contents of log messages, the event broker allows event and system log messages to be emitted from the broker as JSON, with the message arguments in independent JSON properties.

To configure the message format for the event and system syslog facilities, enter the following commands:

solace(configure)# logging
solace(configure/logging)# facility
solace(configure/logging/facility)# {event | system} message-format {text | json}

Where:

event specifies to configure the event syslog facility message format.

system specifies to configure the system syslog facility message format.

text specifies to log message as unstructured text. This is the default value.

json specifies to log message as JSON.

The no form of this command no {event | system} message-format, returns the value to the default.

Log files are not rotated when the format is updated, so at the time of the switch the log file will contain a mix of formats.

JSON Format

When JSON format is enabled, the log message is formatted as JSON (all on a single line) and that JSON is passed to syslog(). The arguments to the log message in Solace PubSub+ Syslog Events is the names of the properties. In addition, the following metadata properties are added at the start of the object in this order:

time — the RFC 3339 timestamp.

solFacility — the Solace name for the syslog facility. Either event or system.

severity — the syslog severity.

host — the hostname.

tag — the event tag.

scope — either VPN, CLIENT, or SYSTEM.

event — the event name, for example CLIENT_CLIENT_DISCONNECT.

vpn — the name of the VPN. Only added if the scope is VPN or CLIENT.

client — the name of the client. Only added if the scope is CLIENT.

raise — an identifier to allow correlation of raise and clear events; also serves to indicate that this event is a raise event. Only added for raise events.

clear — an identifier to allow correlation of raise and clear events; also serves to indicate that this event is a clear event. Only added for clear events.

After the metadata, the arguments follow in the order they are defined. A msg property is added at the end with the event text (with substitutions).

Setting the Maximum Size of JSON Format Syslog Messages

You can set the maximum size of JSON format syslog messages. If the full JSON would exceed that size, properties are dropped or truncated from the end of the message until the value fits within the specified size with an additional property logIncomplete added with the value 1. Only the msg property is truncated, all other properties are dropped if the full value doesn't fit.

Setting a maximum size is useful because the event broker will remove properties or truncate the value of the msg property, but will still produce valid JSON. Downstream processes will generally just truncate the log line, resulting in invalid JSON.

To set the maximum size of JSON format syslog messages, enter the following commands:

solace(configure)# logging
solace(configure/logging)# max-json-message-size <max-size>

Where:

<max-size> is the maximum size in bytes of JSON format syslog messages. If a remote syslog destination is configured, the syslog header is included. The valid range of values is 1024 to 8192. The default value is 8192.

The no version of this command, no max-json-message-size, returns the value to the default.

Viewing Syslog Status

To view the configuration of syslog on your event broker, enter the following command:

solace> show syslog [<name>]

Where:

<name> is the name of the syslog facility destination. Entering no name displays all destinations.

Managing Local Syslog Files

Syslog record files are uncompressed text files that have a <filename>.log.[x] format, where <filename> can be named after a command, event, system, or debug, and x is an integer from 1 to 20 that identifies the archived file number. The active file is named <filename>.log with no numeric suffix.

Retrieving Syslog Record Files

Syslog record files are stored locally as uncompressed text files in the /logs subdirectory on the event broker. To retrieve and view the command log record files, use these CLI commands :

copy — transfers a text file from the /logs subdirectory to an external SFTP or SCP server from the event broker.
Example:
```
solace# copy /logs/command.log sftp://john@10.11.12.13/home/john/solacelogs/
```
more — directly displays the contents of a text file from the /logs subdirectory on the event broker.
Example:
```
solace> more /logs/system.log
```

Logging Capacity

The active syslog record file closes once the file size exceeds 50 MB. Upon closing, a new file opens, and this cycle repeats. The syslog record text files are in the /logs directory and can be retrieved from event brokers. You can view up to 21 syslog record text files for every local syslog facility at any given time, including the active log file. To stay within the 21 file limit, when a new log file is created, the oldest log file is discarded to make space for the new log file. Retention of log files is subject to available disk space. The PubSub+ software event broker caps the space allocated to logs at 1GB.

Syslog record files may contain personal information. To learn more about how to protect this data, see Configuring the Log File Retention Policy.

Directory Maintenance

Because syslog record files are rotated, directory maintenance is not required.

Managing Command Logging

The command logging facility is used to capture information about all event broker commands issued by users through either the Solace CLI or the SEMP interface.

Command Log Record Data Example:

2016-05-02T10:24:51.162-05:00 <local1.info> solace1 admin[12694]: CLI/2        192.168.1.217    admin             10:24:47  10:24:48  ok                              (configure/message-vpn)# max-connections 2000

The information contained in each command log record includes, in order:

log date and time, for example, 2016-05-02T10:24:51.162-05:00
syslog destination and level, for example, <local1.info>
hostname of the event broker, for example, solace1
local user name with local process identifier number or pid in square brackets, for example, admin[12694]
type of event broker interface making the command attempt, for example, CLI or SEMP: CLI/2
IP address of host of source of command attempt, for example, 192.168.1.217
user name assigned to the CLI or SEMP user account
time when command attempt was started, for example, 10:24:47
time when command attempt finished, for example, 10:24:48
description of result, that is: ok, command action status, or reason for failure
command operation attempted: (configure/message-vpn)# max-connections 2000

Configuring Command Logging

To configure the command logging feature on an event broker, which logs user actions and commands to a viewable command log file, enter the following CLI commands:

solace(configure)# logging
solace(configure/logging)# command {cli | semp-mgmt | semp-msgbus| all} mode {shutdown | config-cmds | all-cmds}

Where:

cli, semp-mgmt, semp-msgbus, or all specify the event broker interface to be logged

shutdown turns off the command logging facility for the specified event broker interface

config-cmds specifies log only configuration commands (not show commands) for the specified event broker interface

all-cmds specifies log all commands except for help (help commands are never logged) for the specified event broker interface

The no version of this command (no command {cli | semp | all}) reverts command logging back to the default mode of logging only configuration commands for the specified event broker interface (that is, cli, semp-mgmt, semp-msgbus, or all).

Viewing Command Logging Configuration

To view the configuration of the command logging facility on the event broker, enter the following CLI command:

solace> show logging command

Interpreting Client Disconnect Event Logs

Whenever clients are disconnected from the event broker, the event broker reports the reason for the disconnect through a disconnect event log. This log reports the final statistic summary event for the connection.

SMF Client Disconnect Event Log Example

2013-05-06T18:18:56-0400 <local3.info> solace1 event: CLIENT: CLIENT_CLIENT_DISCONNECT: testVpn testHost/17879/#00010001 Client (82) testHost/17879/#00010001 username testUser WebSessionId (N/A) reason(Peer TCP Closed) final statistics - dp(103, 103, 0, 0, 103, 103, 1557, 2373, 0, 0, 1557, 2373, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) conn(0, 0, 192.168.164.34:58214, CLSWT, 0, 0, 0) zip(0, 0, 0, 0, 0.00, 0.00, 0, 0, 0, 0, 0, 0, 0, 0) web(0, 0, 0, 0, 0, 0, 0), SslVersion(), SslCipher()

MQTT Client Disconnect Event Log Example

2017-07-03T15:59:23+0800 <local3.info> sgdemo1 event: CLIENT: CLIENT_CLIENT_DISCONNECT_MQTT: plant_monitor #mqtt/PLANT_MONITOR_192.168.42.250/71841 Client (1141) #mqtt/PLANT_MONITOR_192.168.42.250/71841 username default ClientId (PLANT_MONITOR_192.168.42.250) reason(Client Disconnect Received) final statistics - dp(2, 1, 1, 0, 3, 1, 80, 4, 238, 0, 318, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) conn(0, 0, 192.168.42.1:20362, ESTAB, 0, 0, 0) mqtt(1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), SslVersion(), SslCipher(), WillSent(0)

The reason entry describes why the client was disconnected:

(Client Disconnect Received)—Connection dropped because a disconnect notification was received.
(Forced Logout)—Administrative disconnect or shutdown action triggered either by a client disconnect, or by replacement of duplicate client connections during authentication.

Replacement of duplicate client connections during authentication is by default enabled on the event broker. To disable, enter the following command: solace(configure)# no authentication replace-duplicate-client-connections.

(Inactive Connection)—Connection dropped due to an inactive network connection.
(Peer Draining Messages Too Slowly)—Connection dropped due to low throughput while the event broker is under stress. During stressful conditions, the event broker drops connections that are moving too slowly to free up system resources.
(Peer Host Unreachable)—The event broker is unable to reach the remote host.
(Peer Refused TCP Connection)—For a connection initiated by the event broker (for example, Message VPN bridging and event broker-to-event broker connections), the peer refused the TCP connection.
(Peer TCP Closed)—The remote end has gracefully closed the TCP connection.
(Peer TCP Reset)—The remote end has ungracefully closed the TCP connection.
(TCP Keep-Alive Failure)—Connection dropped due to a failure on the remote end to respond to a TCP Keepalive probe.
(Too Many TCP Retransmissions)—Connection dropped due to bad throughput.

The dp entry lists the following client dataplane statistics, in order:

Control Messages Received
Control Message Delivered
Topic Messages Received
Topic Messages Delivered
Total Messages Received
Total Message Delivered
Control Bytes Received
Control Bytes Delivered
Topic Bytes Received
Topic Bytes Delivered
Total Bytes Received
Total Bytes Delivered
Current Message Rate at Ingress
Current Message Rate at Egress
Average Message Rate at Ingress
Average Message Rate at Egress
Denied Duplicate Clients

(Messages Discarded at Ingress due to:)

No Subscription Match
Topic Parse Error
Internal Parse Error
Message Too Big

(Queue Discards due to:)

Transmit Congestion

The conn entry lists the following client connection statistics information, in order:

Received Queue Bytes
Send Queue Bytes
Client Address: Port
State
Received Out of Order
Fast Retransmit
Timed Retransmit

The zip entry lists the following client compression statistics information, in order:

Compressed Bytes Received
Compressed Bytes Delivered
Uncompressed Bytes Received
Uncompressed Bytes Delivered
Compression Ratio at Ingress
Compression Ratio at Egress
Current Compressed Byte Rate at Ingress
Current Compressed Byte Rate at Egress
Current Uncompressed Byte Rate at Ingress
Current Uncompressed Byte Rate at Egress
Average Compressed Byte Rate at Ingress
Average Compressed Byte Rate at Egress
Average Uncompressed Byte Rate at Ingress
Average Uncompressed Byte Rate at Egress

The web entry lists the following Web client dataplane statistics information, in order:

Web Messages Received—total number of HTTP requests received by the event broker for the client session.
Web Messages Delivered—total number of HTTP responses sent by the event broker for the client session.
Web Bytes Received—total number of octets in the message bodies of all HTTP requests received by the event broker for the client session.
Web Bytes Delivered—total number of octets in the message bodies of all HTTP responses sent by the event broker for the client session.
Web Out Of Order—approximate number of TCP out-of-order reception events detected by the event broker, which occur while the TCP connection is in use by the client session.
Web Fast Retransmit—approximate number of TCP fast retransmission events initiated by the event broker after receiving duplicate TCP acknowledgments, which occur while the TCP connection is in use by the client session.
Web Timed Retransmit—approximate number of TCP timed retransmission events initiated by the event broker after not receiving TCP acknowledgments, which occur while the TCP connection is in use by the client session.

The mqtt entry lists the following control packet statistics for MQTT clients, in order:

CONNECT packets received
CONNACK packets sent
CONNACK Errors sent
PUBREC—Publish Received responses
PUBLISH—Publish Message packets sent
PUBACK—Publish Acknowledgements Received
PUBACK—Publish Acknowledgements sent
PUBREC—Publish Received packets sent
PUBREL—Publish Release responses received
PUBCOMP—Publish Complete responses sent
SUBSCRIBE—Subscribe to Topics packets received
SUBACK—Subscribe Acknowledgements sent
SUBACK—Subscribe Acknowledgement errors sent
UNSUBSCRIBE—Unsubscribe from Topic packets received
UNSUBSCRIBE—Unsubscribe from Topic packets sent
PINGREQ—PING Requests received
PINGRESP—PING Responses sent
DISCONNECT—Disconnect notifications received
For information on the various MQTT control packets, see 3 MQTT Control Packets in the MQTT Version 3.1.1 Protocol Conformance document.

Where SslVersion lists the TLS/SSL protocol version used.

Where SslCipher lists the cipher suite used.

Interpreting Flow Event Logs

Whenever a flow is closed or unbound from an endpoint on the event broker, the event broker reports the reason for it through a flow event log. This log reports the final statistic summary event for the flow close or unbind, as applicable.

Flow Close Event Log Example

2015-01-08T09:05:40+0000 <local3.info> lab-128-68 event: CLIENT: CLIENT_CLIENT_CLOSE_FLOW: solace dev214/9997/#00010001 Client (6119) dev214/9997/#00010001 username default Pub flow session flow name 99f5f4aa343a4e77bc45f6b3c4db2946 (8048), publisher id 5949, last message id 565, window size 50, final statistics - flow(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 565)

The flow entry lists the following flow close statistics, in order:

Spooling Not Ready
Out of Order Messages
Duplicate Messages
No Eligible Destinations
Spool Over Quota
Queue/Topic-Endpoint Over
Replay Log Over Quota
Maximum Message Usage Exceeded
Maximum Message Size Exceeded
Remote Router Spooling Not Supported
Spool To ADB Fail
Spool To Disk Fail
Errored Message
Queue Not Found
Spool Shutdown Discard
User Profile Deny Guaranteed
No Local Delivery Discard
SMF TTL Exceeded
Publish ACL Denied
Destination Group Error
Not Compatible With Forwarding Mode
Low-Priority-Msg Congestion Discard
Spool File Limit Exceeded
Replication Is Standby Discard
Sync Replication Ineligible Discard
Messages Received

Descriptions for these flow close statistics can be found in Description of Detailed Spooled Message Statistics.

Flow Unbind Event Log Example

CLIENT: CLIENT_CLIENT_UNBIND: RP dev3-185/30296/00030001/349CSHv5cC Client (66) dev3-185/30296/00030001/349CSHv5cC username solclient_1 Unbind to Flow Id (11), ForwardingMode(StoreAndForward), final statistics - flow(255, 0, 0, 0, 0, 2, 0, 0, 2, 0), isActive(No), Reason(Client issued unbind)

The flow entry lists the following flow unbind statistics, in order:

Message Spool Window Size
Used Window
Unacked Messages
Low Message ID AckPending
High Message ID AckPending
Window Closed
Message Redelivered
Message Transport Retransmits
Message Confirmed Delivered Store and Forward
Message Delivered Cut Through

Provide feedback