High Availability in PubSub+ Event Broker: Cloud

PubSub+ Event Broker: Cloud is deployed in high-availability (HA) redundancy groups for fault tolerance. HA redundancy provides 1:1 message broker sparing to increase overall service availability. If one of the message brokers fails or is taken out of service, the other message broker automatically takes over and provides service to the clients that were previously served by the now-out-of-service message broker.

To learn more about HA redundancy, see HA for Software Event Brokers.

PubSub+ Event Broker: Cloud automates this HA redundancy group setup, and makes it easy to use.

HA Concepts

PubSub+ Event Broker: Cloud implements HA using an Active/Standby model with an arbiter node (Monitoring Node) for split-brain detection. This requires three nodes each running the software event broker:

  • Primary node
  • Backup node
  • Monitoring node

The primary and backup nodes both run the software event broker under the messaging node role, while the monitoring node runs it under the monitoring node role. Each of their respective roles is fixed by the configuration and never changes. The HA group is fronted by a network load balancer that routes traffic to-and-from the active node in the HA group (either the primary or backup).

When in operation, the messaging nodes will assume one of these Active/Standby roles: Primary or Backup. At any one time, one node is the primary and the other is the backup.

Upon a failover, connections to the broker are switched over from the Primary to the Backup node automatically.

HA in Public and Private Clouds

To ensure that high-availability group is adequately provisioned, pods run on different worker nodes. Additionally, the pods can be spread over multiple Availability Zones (AZ) when available. The following diagram shows a Kubernetes cluster that has worker nodes over three availability zones. The Cloud-Agent will schedule the Messaging nodes over two AZ and the monitor node on a third AZ. For each HA service, the primary pod is deployed in one AZ, the backup pod in a second AZ, and the monitoring pod in a third AZ. This guarantees that pods for the same HA service are not running on the same hardware.

Similarly, when deploying a HA group in virtual private clouds such as AWS, there are two Network Topologies available.

  1. For regions with three or more Availability Zones (AZ):

  2. For regions with two Availability Zones:

Connecting to a Cloud HA Group

Typically, applications using HA would have to provide a host list: one IP address for the primary node and another for the backup node. However, this approach (providing hosts list) will not work for 3rd-party messaging APIs, so PubSub+ Event Broker: Cloud uses a single DNS entry for applications to use (behind a load balancer) abstracting away the switchover between primary and backup in the event of a failure.

HA and Service Types

The following service types deploy an HA redundancy group by default:

  • Professional (Standard account)
  • Enterprise (Enterprise account)

PubSub+ Cloud automates all of the configuration and setup when you create your event broker service. Once the service is created, applications can use the DNS name entry provided in the connectivity tab in the console.

HA Link Security

When a new enterprise event broker service is created, the communication between the primary and backup event brokers are encrypted by default, including the HA mate link and config-sync. You can override the default HA Mate link encryption to plain text through the advanced option in the console. Overriding the default HA mate link encryption to plain text may be useful if you require maximum performance, and are willing to trust the security restrictions of the VPC in the cloud providers or on-premises. Config-sync will always remain encrypted.

If you have an existing event broker service without encryption, you can encrypt it, including its HA mate link and config sync link through the console or the REST API. In the console, you can easily differentiate between the encrypted services and ones that are not; when the mate-link encryption is disabled, a warning icon is displayed on the event broker service's status screen.

Modifying Mate-link Encryption

The HA mate-link is encrypted by default. When creating an event broker service, you can override the default encryption to plain text through the Advanced Connection Options available in the console.

Once the service is created, the status of the mate-link encryption is displayed on the service's Status tab.

You can also modify HA mate-link encryption after creating a service. To do so, follow the steps below:

  1. In Cluster Manager, select a service, and click the Manage > Advanced Options.

  2. In the next screen, select the Disable or Enable to modify the encryption.