System Status

Solace Insights Monitors for Datadog Reference provides information to help you understand how to read and filter the information provided by the monitors offered by your Insights Datadog account.

These monitors track system-level status and health, including bridge connectivity, cache instance and cluster status, guaranteed messaging service status, message queue bindings, REST Delivery Point status, and message spool resources.

Monitor Name	Monitor Type	Status	Default Severity Configured	Details
Bridge Down Observed	Status	Active	Warning	The Message VPN Bridge (bridge) is down. Predicted Impact: If the bridge remains down, messages can no longer be transmitted between the bridged event brokers. Recommended Actions: Verify that your service bridge configuration and network connectivity of the bridged event brokers are correct. For mitigation recommendations, see VPN_BRIDGING_LINK_DOWN in the Syslog Events Reference or Contact Solace for assistance if required.
Bridge Down Sustained	Status	Active	Alert	The bridge has been down for a sustained period. Predicted Impact: If the Bridge remains down, messages can no longer be transmitted between the bridged event brokers. Recommended Actions: Verify your service bridge configuration and network connectivity of the bridged event brokers. For mitigation recommendations, see VPN_BRIDGING_LINK_DOWN in the Syslog Events Reference. Contact Solace for assistance if required.
Cache Cluster Status Down	Status	Active	Alert	All Cache Instances in the cluster are `DOWN`. Predicted Impact: Messages for this cluster are being lost and cache lookups are unavailable. Recommended Actions: Recover the affected cache instance so that it's connected and operational. For mitigation recommendations, see the related VPN_SOLCACHE_* logs in the Syslog Events Reference. Contact Solace for assistance if required.
Cache Instance Memory - Out of Sync	Metric	Active	Warning	The cache instance memory utilization is not synchronized. Predicted Impact: There is an elevated risk of message loss until the cache instance is recovered. Recommended Actions: Check the memory utilization for each cache instance within the cache cluster. If the memory utilization difference persists, then restart the cache instance which has been identified as out of sync. For mitigation recommendations, see the related VPN_SOLCACHE_* logs in the Syslog Events Reference. Contact Solace for assistance if required.
Cache Instance Status Down Observed	Status	Active	Warning	The cache instance is down. Predicted Impact: There is an elevated risk of message loss until the cache instance is recovered. Recommended Actions: Recover the affected cache instance so that it's connected and operational. For mitigation recommendations, see the related VPN_SOLCACHE_* logs in Syslog Events Reference. Contact Solace for assistance if required.
Cache Instance Status Down Sustained	Status	Active	Alert	Cache Instance has been down for a sustained period. Predicted Impact: There is an elevated risk of message loss until the cache instance is recovered. Recommended Actions: Recover the affected cache instance so that it's connected and operational. For mitigation recommendations, see the related VPN_SOLCACHE_* logs in the Syslog Events Reference. Contact Solace for assistance if required.
Dead message queue unbound and with messages	Metric	Active	No Default Severity	A dead message queue (DMQ) has messages but no clients are connected to process them. Predicted Impact: There is an elevated risk that if the queue fills up, messages are rejected. Recommended Actions: Verify the queue connections status in Cluster Manager from the Solace Cloud Console.
Guaranteed Messaging Service Interruption Observed	Status	Active	Alert	The event broker's message spool disk has not been in the `AD_Active` state for the last minute. This means the message spool disk has not been bound to the primary event broker for the last minute. This may be due to normal event broker activity. Predicted Impact: There is an elevated risk that applications may not be able to send or receive Guaranteed Messages. Recommended Actions: This alert is sent for informational purposes and no action is required. If the situation degrades, the Guaranteed Message Service Interruption Sustained monitor will trigger an alert, and you can take the actions described for that monitor.
Guaranteed Messaging Standby Status Down Sustained	Status	Active	Alert	Persistent messaging (Guaranteed Messaging) redundancy has decreased for a sustained period. Predicted Impact: There is an elevated risk for a service outage. Recommended Actions: For mitigation recommendations, see the SYSTEM_AD_* logs in the Syslog Events Reference. Contact Solace for assistance if required.
Guaranteed Messaging Service Interruption Sustained	Status	Active	Alert	The event broker's message spool disk has not been in the `AD_Active` state for the last five minutes. This means the message spool disk has not been bound to the primary event broker for the last five minutes. The event broker may be experiencing a Guaranteed Messaging service interruption. Predicted Impact: There is an elevated risk that applications may not be able to send or receive Guaranteed Messages. Recommended Actions: For mitigation recommendations, see the SYSTEM_AD_* logs in the Syslog Events Reference. For more information, Contact Solace.
Guaranteed Messaging Standby Status Down Observed	Status	Active	Warning	Guaranteed messaging redundancy has decreased. Predicted Impact: There is an elevated risk for a service outage. Recommended Actions: For mitigation recommendations, see the SYSTEM_AD_* logs in the Syslog Events Reference. For more information, Contact Solace.
Message queue unbound and with messages	Metric	Active	No Default Severity	A message queue has messages but no clients (bound) are connected to process them. Predicted Impact: If the queue fills up, messages may be rejected. Recommended Actions: Verify the queue connections status in Cluster Manager from the Solace Cloud.
RDP Status Down Observed	Metric	Active		A REST Delivery Point (RDP) connection has reported an unsuccessful status. Predicted Impact: If the problem persists this may result in failure to deliver messages. Recommended Actions: Verify the RDP connectivity to the target endpoint and the service that provides it. For more information, see Creating REST Delivery Points.
System Message Spool - Active Disk	Metric	Active	Alert or Warning or Recovery	A system message spool active disk partition utilization threshold breach. Active disk partition utilization exceeds the monitor's threshold. Solace support has been notified and is working to resolve the situation. This could be a symptom of a mismatch between the volume of messages being published and the available consumers or speed of consumers. Predicted Impact: If the partition reaches 100% utilization (disk space exhausted), no new Guaranteed Messages will be accepted by the event broker. Recommended Actions: Review your consumers and explore if there are issues with the speed or the number of consumers. Review publishers to determine if there is an overall increase in incoming messages. Consider adding more consumers or addressing any slow consumers.
System Message Spool - Message Count	Metric	Active	Alert or Warning	A message spool message count utilization threshold breach. Predicted Impact: If this resource is 100% of capacity (resources exhausted), no new Guaranteed Messages will be accepted by the event broker. Recommended Actions: Check that the endpoints with the most messages enqueued. Contact the support or development team responsible for the client applications that consume messages from these endpoints.
System Message Spool - Queue Subscription	Metric	Active	Alert or Warning or Recovery	The system message spool queue subscription utilization threshold has been breached, exceeding the monitor thresholds. Predicted Impact: You may not be able to add more persistent subscriptions. Recommended Actions: Verify the endpoints with the subscriptions, and contact the maintainers of the client applications responsible for these endpoints.
System Message Spool - Unacknowledged Messages	Metric	Active	Alert or Warning	System message spool unacknowledged message utilization threshold breach. The number of delivered but not yet acknowledged Guaranteed Messages exceed the monitor thresholds that are set. Predicted Impact: New Guaranteed Messages may be rejected. Recommended Actions: Identify the Guaranteed Messaging flows with large number of unacknowledged messages and disconnect the associated clients.
System Storage - Storage Element	Metric	Active	Alert or Warning or Recovery	An event broker in a Customer-Controlled Cluster has used a certain percentage of a given storage element. A storage element in Solace Cloud is a persistent volume mounted as a system storage device for an event broker. The storage element is used for various functions such as the message spool, log storage, runtime operation. Predicted Impact: If the storage element fills up, the event broker may begin returning negative acknowledgments to publishing clients. Recommended Actions: For mitigation recommendations, see the SYSTEM_CHASSIS_DISK_UTILIZATION_HIGH and SYSTEM_CHASSIS_DISK_UTILIZATION_MAX logs in the Syslog Events Reference , or consider increasing the Message Spool size. For more information, see Configuring Message Spool Sizes. For more information, Contact Solace.
System Storage - Storage Element(root or config)	Metric	Active	Alert or Warning or Recovery	An event broker in a Customer-Controlled Cluster has used a certain percentage of a given storage element. A storage element in Solace Cloud is a persistent volume mounted as a system storage device for an event broker. The storage element is used for various functions such as the message spool, log storage, runtime operation. Predicted Impact: If the storage element fills up, the event broker may not be able to save new system files or configurations. Recommended Actions: For mitigation recommendations, see the SYSTEM_CHASSIS_DISK_UTILIZATION_HIGH and SYSTEM_CHASSIS_DISK_UTILIZATION_MAX logs in the Syslog Events Reference . For more information, Contact Solace.
Topic endpoint unbound and with messages	Metric	Active	No Default Severity	A topic endpoint has messages but no clients are connected to process them. Predicted Impact: If the topic endpoint fills up, messages may be rejected. Recommended Actions: Verify the topic endpoint connection status in Cluster Manager in the Cloud Console.

Provide feedback