It's recommended that you monitor latency across your messaging infrastructure.
- End-to-end latency across your messaging infrastructure can be measured by building a publish/subscribe tool that takes a timestamp, publishes it in a message to a queue, receives the message from the queue, and calculates the difference between the current time and the timestamp in the received message.
- End-to-end system latency should be measured in all environments, not just production.
- If timestamps are needed at various points along a message route, it should be noted that there may be inaccuracies due to time synchronization and drift issues between hosts.
Example test setup - also applicable to the software event broker
Below is a summary of steps followed by an application to perform end-to-end latency monitoring:
- Monitoring App1 sends request messages at a rate of 10 msgs/sec to its local event broker including timestamp values in each message.
- Request messages are routed to remote the remote event broker via the Multi-Node Routing feature.
- Monitoring App2 receives the request messages and replies immediately back to the requests.
- The reply messages are routed back the App2’s local event broker.
- Monitoring App1 receives the reply messages and compares the start and end timestamp value.
- Monitoring App1 should also subscribe to Syslog event messages over the message-bus and poll stats via SEMP on a separate thread. Optionally this can be done by a separate application or monitoring tool.
Advantages of using this monitoring approach:
- Latency measurement is accurate as timestamp values are compared on the same monitoring app server. Therefore time clocks don’t need to be synchronized.
- This method provides end-to-end latency monitoring.
- Differences in latencies can be correlated to events raised on the message bus, if any, or stats that are queried using SEMP.
- Additionally, per-connection latency stats can be retrieved by requesting the client connection wide stats.