Installing PubSub+ Cloud in Amazon Elastic Kubernetes Service (EKS)

Amazon Elastic Kubernetes Service (Amazon EKS) gives you (the customer) the flexibility to start, run, and scale Kubernetes applications in the AWS cloud or on-premises. Amazon EKS helps you provide highly-available and secure clusters and automates key tasks such as patching, node provisioning, and updates. For more information about EKS, see the Amazon EKS documentation.

Depending on the Kubernetes distribution you choose, there are a number of environment-specific steps that you must perform to install PubSub+ Cloud.

Before you perform the environment-specific steps described below, ensure that you review and fulfill the general requirements listed in Common Kubernetes Prerequisites.

1. Create a Kubernetes cluster. For customer-owned deployments, you are responsible for the set up of the Kubernetes cluster and the maintenance and operation of the cluster. The following information can help you to understand the requirements of that Kubernetes cluster that you create:
   • The Kubernetes cluster must fulfill the Amazon EKS Prerequisites
   • Review and choose from various considerations to deploy your EKS cluster.
   • One way to create and understand the requirements for the Kubernetes EKS cluster is to use the examples (Terraform module and deployment scripts) available from Solace.

   Available from Solace are sample scripts and Terraform modules you can use as reference example to understand what is required in your Kubernetes cluster. The example is provided as-is. You (the customer) can modify the files as required to create your Kubernetes cluster. If you choose to do so, then you are responsible to maintain and modify the files for your deployment. For more information, contact Solace.

Amazon EKS Prerequisites

The following are the technical prerequisites for an Amazon EKS deployment to deploy event broker services:

VPC Security Group

Before you begin, you must open an AWS support ticket and request an increase to the Rules per VPC Security Group to 200 for the region you intend to deploy your EKS. This increase is required for event broker services to support the default protocols, which are as follows:

• SSH
• WEB-messaging
• SEMP
• AMQP TLS
• MQTT web tls
• MQTT TLS
• REST
• SMF/TLS
• Load Balancer

Permissions

An AWS account with the following permissions. These permissions are required only by the individual when deployment is done using a Terraform module:

• All the permissions that are required to create and manage the EKS cluster (eksClusterRole).
• Permission to create IAM roles and IAM policies in the EKS cluster. These permissions are required by the Terraform module. The example module creates these IAM roles and policies that are used by the EKS cluster, and the following permissions to create and manage resources in the EKS cluster:

  IAM Role

  Gives permissions to the following:

  • EKS cluster control plane
  • EKS cluster worker nodes
  • EKS Load balancer controller
  • EKS auto-scaler

  IAM Policy

  Creates a set of permissions that the is required by the following in the deployment:

  • EKS load balancer controller (for more information, see the Amazon load balancer controller installation document)
  • Auto-scaler (for more information about creating this policy, see Cluster Autoscaler in the Amazon documentation)

  All EC2 resources

  The Kubernetes and Terraform modules require this permission to access tags and the metadata of resources. The autoscaler and Load Balancer provisioning look at what security group each instance has, and modifies the security groups to add rules for the load balancer services.

  VPC

  The Terraform module requires this permission to create the VPC. The Kubernetes module also requires this permission to scan VPC and subnets to retrieve networking parameters.

  Elastic Load Balancers

  Permission is required for the Kubernetes module to create ELBs for the Load Balancer services. The Load Balancer services requires rules that are added in the Security Groups mentioned below. Additional permissions required by the AWS Load Balancer Controller and must be updated to match those specified at https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json.

  EBS

  Permission is required for Kubernetes dynamic PVCS that require access to EBS in order to dynamically create values and attach on the correct host. The host also requires EC2 access as described above.

  Security Groups

  The Kubernetes module requires this permission to create security groups. The load balancer services also requires rules added as well.

  Routing tables

  The Terraform module requires this permission to create routing tables.

  Internet Gateways

  Terraform module requires this permission to create an Internet gateway.

  Elastic IPs

  The Terraform module requires this permission to attach Elastic IP addresses (EIPs) to the NAT gateway.

  NAT Gateways

  The Terraform module requires this permission to create the NAT gateway and attach EIPs to the NAT gateway.

  OIDC Provider

  The Terraform module requires permission to create an OIDC provider that will be used to authenticate Kubernetes modules against IAM roles. The Auto-scaler and AWS Load Balancer Controller are two of the Kubernetes modules that use OIDC to authenticate the IAM role.

Networking

You must create an Elastic IP address (EIP) for each NAT Gateway that you intend to use with the following considerations:

• The EIPs for the NAT gateways must be created upfront.
• Solace recommends two EIPs are created. A minimum of one EIP allocation ID is required.

Considerations for Deploying PubSub+ Cloud on an Amazon EKS Cluster

You (the customer) should consider the following limitations and recommendations regarding your Amazon EKS cluster for a private data center deployment of PubSub+ Cloud:

• A minimum size of /24 is required for an EKS cluster that's dedicated to event broker services. A larger size is required if the cluster contains other services. For more information, see IP Range.

• Solace recommends that you have two NAT Gateways and two EIPs for redundancy. You can choose to use one NAT Gateway with one EIP with the consideration that all event broker service features [e.g., VPN bridges, Dynamic Message Routing (DMR), Disaster Recovery (DR), REST Destination Points (RDP)] that rely on external connections will fail to function if the zone of the NAT Gateways fails.
• Solace recommends that you have a least two Bastion hosts, and that these hosts are spread out evenly over the three Availability zones. This recommendation allows you to remotely access the deployment should a zone fail. It's important to note that remote access to your deployment if a zone fails is not a requirement; you can choose to use the minimum of one Bastion host instead.

Considerations for Deployments in China Regions

Additional considerations are required if the private region you deploy to is within China.

• You must have a separate Amazon Web Services (China) Account, which is a set of credentials that are distinct and separate from the Amazon Web Services Global Accounts. To register an account, go to http://www.amazonaws.cn.

• Customer-Controlled Regions in China require that you provide a custom domain name registered in China (for example, *.mycompany.cn) with certificates issued for that domain. Event broker services are accessed using the custom domain name rather than *.messaging.solace.cloud. For more information, see Deployments in China.

• Instead of the GCP registry, you must use the Azure China registry. The Azure China registry uses a different secret that's required when you later deploy the Mission Control Agent. For example, run the following command where the <username> and <password> is provided by Solace.

  kubectl create secret docker-registry cn-reg-secret --namespace kube-system \ 
  --docker-server=solacecloud.azurecr.cn --docker-username=<username>  --docker-password=<password>

All other EKS clusters specifications are the same as described in EKS Cluster Specifications.

EKS Cluster Specifications

Before you (the customer) install the Mission Control Agent, you must configure the EKS cluster with the technical specifications listed in the following sections:

• Node Groups
• Instance Type Requirements
• Bastion Host
• Storage Class
• Networking
• NAT Gateway
• Load Balancer
• IP Range
• Autoscaling

For more detailed information about using Amazon EKS, see the User Guide on the Amazon EKS documentation site.

Node Groups

The Kubernetes cluster autoscaler must use two node groups. The nodegroups can be configured to start from zero instances, which means that it has a full set of node groups for each scaling tier without requiring that instances run in each.

The node groups must be configured to start at 0 . Hints should be provided to the autoscaler as to which labels and taints are set on the worker nodes by using stags on the auto-scaling group (ASG). There is no mechanism available to tag the ASG when you create the node groups, but can be accomplished as described in Managed Nodes Scale to Zero and Cluster Autoscaler does not start new nodes when Taints and NodeSelector are used in EKS.

Instance Type Requirements

Because of the additional resources required to run Kubernetes, the instance types that are required for some of the scaling tiers are larger than their instance-based cousins. the following are the instance size type requirements for an EKS. For details about the core and RAM requirements for each scaling tier, see Resource Requirements for Kubernetes and Default Port Configuration.

Scaling Tier	Instance Type Required
Monitor	T3.medium
Developer	R5.large
Enterprise 250	R5.large
Enterprise 1K	R5.large
Enterprise 5K	R5.xlarge
Enterprise 10K	R5.xlarge
Enterprise 50K	R5.2xlarge
Enterprise 100K	R5.2xlarge

Bastion Host

Solace recommends that you have at least two Bastion hosts, and that these hosts are spread out evenly over the three Availability zones. This allows you to remotely access your deployment should a zone fail; if remote access to your deployment if a zone fails is not a requirement, you can choose to use the minimum of one Bastion host instead to manage the EKS cluster. The Bastion host is deployed using a minimal VM that runs only an SSH server.

Storage Class

The EKS storage class (type) can use either GP2 or GP3. Consider the following when choosing which storage class to use:

• With GP2, the performance improves as the size of the disk increases. In cases where the your disk size is greater than 1 TB, using GP2 is a better option than GP3.

• With GP3, the default performance is 3k IOPS regardless of disk size. GP3 is recommended for disk sizes that are less than 1TB. For cases where the disk size is greater than 1 TB, GP2 is the better choice unless you configure the GP3 storage class with a provisioned IOPS that's higher than 3k. It's important to note that if you increase the IOPS on the storage class, then all event broker services get the same IOPS. For example, if you increase to 6k IOPS, then all event broker services get 6k IOPS disks and you may incur an extra 3k IOPs costs as a result of doing this.

It's important remember that the disk size (the size of the EBS volume) is twice the Message Spool size specified when you create an event broker service. For example, if you configure an event broker service to use a Message Spool of 500 GB, you require a 1 TB disk size and this should be a considered as part of your planning when you create the Kubernetes cluster.

To deploy PubSub+ Cloud, the StorageClass in EKS must be configured to use the WaitForFirstConsumer binding mode (volumeBindingMode). To support scale-up, the StorageClass must contain the allowVolumeExpansion property, and have it set to "true". If an optional AWS KMS (CMK) key is used, that key must be provided to Solace. The KMS key can be applied to the cluster during or after creation of the Kubernetes cluster. You should always use XFS  as the filesystem type (fsType).

The properties of your StorageClass yaml should be similar to the following:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: gp2
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
parameters:
  encrypted: "true"
  fsType: xfs
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

For more information about creating a storage class for EKS, see Supported Storage Solutions on the Amazon EKS User Guide site.

Networking

To spread an event broker service, or rather the event brokers over the Availability Zones (AZ), an anti-pod affinity should be used. When the region has multiple availability zones, the topologyKey key should be set to kubernetes.io/zone, whereas when AZs not are available, set it to topology.kubernetes.io/hostname.

NAT Gateway

You require one Elastic IP (EIP) for each NAT gateway for your cluster. Solace recommends that you have two Elastic IPs (and two NAT gateways) for a production system.

You can have up to three EIPSs and NAT gateways, which allows you to have multi-AZ NAT redundancy . This requires that you have three EIPs. These NAT EIPs must be created upfront. If you use a Terafom module, ensure you use the EIPs.

Load Balancer

PubSub+ Cloud requires the deployment of the AWS Load Balancer Controller to create Network Load Balancers (NLB) to front event broker services. You can find instructions for deploying the AWS Load Balancer Controlled at https://kubernetes-sigs.github.io/aws-load-balancer-controller

Solace configures the NLBs used in PubSub+ Cloud with IP targets and cross-zone enabled. This results in the fastest possible failover times.

Solace uses the following service annotations to configure the NLB:

service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing # this one is removed for internal (private) services
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: '2'
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: '2'
service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: '5550'
service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http
service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /health-check/guaranteed-active
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: '6'
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: '10'

 

If you configured your EKS cluster using Solace's custom AWS Load Balancer configurations and require information about the custom AWS Load Blanacer see Custom AWS Load Balancer for PubSub+ Cloud.

IP Range

A minimum size of /24 is required for an EKS cluster that's dedicated to event broker services. A larger size is required if the cluster contains other services.

Amazon's VPC Container Network Interface (CNI) allocates IPs directly from the VPC’s subnets. The number of IPs that are allocated is directly proportional to the number of worker nodes, which is also proportional to the number of event broker services.

The calculations below are based on custom settings for WARM_IP_TARGET and WARM_ENI_TARGET:

kubectl set env ds aws-node -n kube-system WARM_IP_TARGET=1     
kubectl set env ds aws-node -n kube-system WARM_ENI_TARGET=0

Details about these settings are available in the Amazon Kubernetes VPC CNI documentation on GitHub.

Event Broker Service IP Requirements

The high-availability (HA) event broker service has three event brokers (primary, backup, and monitor) which require five IPs per subnet: four per service, plus one for the internal network load balancer. This totals 15 required IPs.

A standalone service requires five IPs in one subnet, and one IP in each of the other two subnets.

Private Subnet IP Requirements

For private subnets, both HA and standalone services require an initial 8+13 IP addresses for infrastructure and overhead. The value of 13 accounts for the worker nodes and overhead required for the underlying default infrastructure.

Calculate the IP requirements for the event broker services in a private subnet with the following formulas:

8 + 13 + (4 x (number of HA services)) + (4 x (number of SA services)) / 3 + (total number of services)

Public Subnet IP Requirements

For public subnets, both HA and standalone services require an initial 8+2 IP addresses for infrastructure and overhead. The value of two accounts for the required NAT gateway and Bastion host.

Calculate the IP requirements for the event broker service in a public subnet with the following formula:

8 + 2 + (number of services)

CIDR Calculator

You can use the Solace provided downloadable excel-based CIDR calculator to calculate your CIDR requirements.

Examples

Below are the calculations for the VPC and subnets for an EKS cluster containing six standalone event broker services and 1 HA event broker service.

Private subnets

8 + 3 + (4 x 1) + (4 x 6) / 3 + 7 = 40 IPs

Public subnets

8 + 2 + 7 = 17 IPs

These calculations result in the following VPC and Subnets:

VPC: 10.10.0.0/23

Public Subnet 1: 10.10.0.0/27
Public Subnet 2: 10.10.0.32/27
Public Subnet 3: 10.10.0.64/27

Private Subnet 1: 10.10.0.128/25
Public Subnet 2: 10.10.1.0/25
Public Subnet 3: 10.10.1.128/25

Limitations

Currently, the limit for event broker services in a cluster using the standard aws-load-balancer-controller is 11 due to security group rule limits. You can avoid this limitation by using a modified aws-load-balancer-controller provided by Solace.

Autoscaling

Your cluster requires autoscaling to provide the appropriate level of available resources for your event broker services as their demands change. Solace recommends using the Kubernetes Cluster Autoscaler, which you can find in the Kuberenetes GitHub repository at: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler.

See the Autoscaling documentation on the Amazon EKS documentation site for information about implementing a Cluster Autoscaler.