Mission Control Agent

The Mission Control Agent is required to deploy and manage PubSub+ Cloud event broker services. The Mission Control Agent performs these functions:

• Managing the lifecycle and configuration of event broker services in the deployment, including creation, deletion, updates, and configuration options on all event broker services in the data center

• Gathering system-level diagnostic information of the event broker services to send to the PubSub+ Home Cloud to allow for rapid response to address any issues. The information is provided solely to help with the recovery of event broker services. For more information, see Information Exchanged Between the PubSub+ Home Cloud and the Mission Control Agent

• Managing the lifecycle of the monitoring agents, which send metrics and monitoring data from event broker service to the central monitoring service. This is required for event broker services to be monitored and necessary for PubSub+ Cloud to function. For more information, see Information Exchanged Between the Mission Control Agent and the Centralized Monitoring Service.

The Mission Control Agent resides in the same network where the event broker services are deployed. In the following architecture diagram, the red box highlights the Mission Control Agent.

The security measures applied to communication between the Mission Control Agent and the PubSub+ Home Cloud include:

• The Mission Control Agent always initiates the connection to the Home Cloud. It does not listen, nor accept incoming connections.

• The Mission Control Agent never transmits any messaging data, schema information, or personal account information.

• The Mission Control Agent receives management data, consisting of configuration information for the event broker services, and sends only metadata related to orchestrating an event broker service and monitoring data related to system-level checks on the event broker service components. For more information about the data shared, see Information Exchanged Between the PubSub+ Home Cloud and the Mission Control Agent

• The data exchanged between the Mission Control Agent consists of information collected only from the event broker services that the Mission Control Agent manages.

• The Mission Control Agent authenticates with the Home Cloud, allowing it to connect securely to exchange information. The Mission Control Agent does not require a public IP address.

• The information sent between the Mission Control Agent and the Home Cloud is encrypted and can't be read by the cloud service providers (Amazon, Google, or Microsoft).

The Mission Control Agent requires a minimal number of permissions to run within the customer's network to manage the lifecycle and configuration of event broker services.

For more information about Mission Control Agent security, see the following sections:

• Mission Control Agent Installation
• Information Exchanged Between the PubSub+ Home Cloud and the Mission Control Agent
• Information Exchanged Between the Mission Control Agent and the Centralized Monitoring Service
• Mission Control Agent Communication within the Kubernetes Cluster
• Mission Control Agent Connectivity
• Permissions Required for the Mission Control Agent

Mission Control Agent Installation

During installation, the Mission Control Agent is bootstrapped with the configuration it needs to manage event broker services in the customer's environment. This configuration includes:

• for private cloud deployments:
  • restricted IaaS credentials for the customer's virtual network or VPC
  • IaaS infrastructure details, such as the names of the virtual network or VPC and subnets
  • SSH Key Pair (.pem file) used to configure cloud instances or virtual machines for event broker services
• for Kubernetes deployments:
  • the data center ID and target namespace
  • restricted service account credentials. This service-account is bound to a custom role with the minimum necessary permissions, which is scoped to the target namespace.
  • infrastructure details, such as the names of the primary, backup, and monitoring zones, configuration details for Datadog, and storage and load balancer parameters

Additionally, the customer data center is registered with the Home Cloud and is locked to the customer account.

Information Exchanged Between the PubSub+ Home Cloud and the Mission Control Agent

The Mission Control Agent authenticates with the PubSub+ Home Cloud with basic authentication, allowing it to connect securely to the Home Cloud to exchange information. The Mission Control Agent always initiates of the connection with the Home Cloud. The PubSub+ Home Cloud never initiates the call.

The Mission Control Agent never transmits messaging data or any personal account information and never exchange any access keys or credential information of any user. It send only control plane (management) information to the Home Cloud.

The Mission Control Agent receives management data, consisting of configuration information for the event broker services, and shares only metadata related to orchestrating an event broker service and monitoring data related to system-level checks on the event broker service components that the Mission Control Agent has permission to access.

The information that is exchanged between the Home Cloud and the Mission Control Agent includes:

• Event broker services — The metadata required to configure an event broker service, including cloud instance, IaaS configuration details, configuration specifications (e.g., ARM templates, CloudFormation stacks), and upgrade the Mission Control Agent. This includes responses codes and status information for Home Cloud initiated actions (upgrades, service creation and deletion, etc.), confirmation as to whether the action completed as intended are collected.

• Heartbeats — Health checks for various components of the event broker services are logged.

• Response codes and status — The Home Cloud initiated actions (upgrades, service creation and deletion, etc.), confirmation as to whether the action completed as intended are collected.

• Commands — Only high-level commands to create, configure, and manage event broker services are exchanged.

• DNS configuration — The Mission Control Agent coordinates DNS configuration with the Home Cloud. This information is necessary to communicate with the event broker services.

• Certificates — The Home Cloud provides the signed URL to the Mission Control Agent to allow the Mission Control Agent to load the server certificates to the event broker services.

• Diagnostic information — The Mission Control Agent collects and shares diagnostic information to assist with incident handling. No personal information is collected as part of the diagnostic information.

For more detailed information about the data exchanged between Mission Control Agent and the Home Cloud, contact Solace.

Information Exchanged Between the Mission Control Agent and the Centralized Monitoring Service

Solace uses the Datadog cloud application for its central monitoring service component. The Datadog agent is the monitoring component that resides in a deployment. There is one Datadog agent per event broker service. For more details, see Deployment Architecture for Kubernetes.

The Mission Control Agent configures the Datadog agents for monitoring, but no other information is exchanged. The Datadog agents collect and send monitoring information (logs and metrics) to the central monitoring service about an event broker service and metrics about the Mission Control Agent. For more details about the central monitoring service, see Central Monitoring Service and Datadog Agents.

Mission Control Agent Communication within the Kubernetes Cluster

The Mission Control Agent communicates with various Kubernetes cluster components, which includes the event broker services deployed in the cluster.

• The communication with the event broker services consists of information exchanged to manage their lifecycle and configuration operations that includes creating, deleting, updating, and configuring event broker services. This communication uses a combination of SEMPv1, SEMPv2, and Solace CLI commands (via Kubernetes exec mode).

• Communication within the Kubernetes cluster is for Kubernetes API and kubelets.

The following table shows the destination ports that are used within the Kubernetes cluster (source ports are ephemeral):

Port	Requirement Within Kubernetes Cluster
1943	This port is required for communication between the event broker services and the Mission Control Agent to manage lifecycle operations. External SEMP requests are that are received on port 943 are forwarded to 1943 to the containers for the event broker services.
443	This standard port is required for interaction between the Mission Control Agent and the Kubernetes API that is required for control plane operations to manage event broker services.
6443	This standard port is required for communication of between the Mission Control Agent and kubelets (including exec kubectl) required for control plane operations to manage event broker services.

Port 1943 must be enabled within the Kubernetes cluster, however Port 1943 does not need to be enabled for incoming connections to the Kubernetes cluster for event broker services.

Mission Control Agent Connectivity

Connectivity for the Mission Control Agent does not require a public IP address and the Mission Control Agent always initiates the connection to the PubSub+ Home Cloud. The Mission Control Agent communicates with the Home Cloud in the following ways:

• securely over port 55443 using the Solace Message Format (SMF) protocol over TCP

• Securely over HTTP port 443 during installation and bootstrapping. This is also how the docker image for the Mission Control Agent is retrieved from the Solace registry for Kubernetes installations

The Docker images that are used in deployment are available from the Solace Container Registry (gcr.io) using port 443 (secure port). Having the Docker images available in the gcr.io is ideal if your security policies require that all images are scanned prior to deployment.

Permissions Required for the Mission Control Agent

The Mission Control Agent requires a service account with specific permissions. The service account requires a minimal set of permissions that permit the Mission Control Agent to manage and configure event broker services and communicate with the PubSub+ Home Cloud.

The following permissions are required for the Mission Control Agent:

The Mission Control Agent is assigned a service account called cloud-agent; this account is created automatically by the Helm chart.

This service account is bound to a role called cloud-agent-role, which is scoped to the target namespace. Solace does not support the integration of event broker services with service meshes, such as Itsio, Cillium, and Linkerd. If your cluster has a service mesh, this namespace must be excluded from it. The service account is also bound to the Docker Registry secret which gives it access to Solace's enterprise Docker images.

The cloud-agent-role gives the Mission Control Agent permissions for the following namespace resources:

Secrets

The Mission Control Agent needs to create, update, and delete secrets for the event broker service it manages.

Services

The Mission Control Agent needs to create, update, and delete services to expose theevent broker serviceTCP ports to its clients.

configmaps

The Mission Control Agent needs to create, update, and delete configmaps for the event broker service it manages.

Pods

The Mission Control Agent needs to update and delete pods for the event broker service it manages.

Pods/Exec

The Mission Control Agent needs to execute commands in the event broker service's pods for certain operations such as in-service upgrades and configuring the monitoring agent.

Persistent Volume Claims

The Mission Control Agent needs to update and delete PVCs for the event broker service it manages.

Events

The Mission Control Agent needs to retrieve Events generated by Statefulsets, Jobs, and Services to report scheduling errors and Service creation failures.

Statefulsets

The Mission Control Agent uses Statefulsets as controllers for the event broker service pods. It needs to create, update and delete Statefulsets as part of managing the lifecycle of the event broker services.

Deployments

The Mission Control Agent needs deployment permissions to perform self-upgrades and to create, upgrade, and delete distributed tracing deployments.

Jobs

The Mission Control Agent needs to create, monitor, and delete Jobs to perform schema migration during upscaling operations. This is accomplished by launching a Pod via the Job controller.

Pod Disruption Budgets

The Mission Control Agent creates a Pod Disruption Budget (PDB) for each software event broker that it deploys. It also manages the PDBs afterward.

PDBs are required by Kubernetes worker node upgrades to ensure that event broker services remain operational during Kubernetes rolling upgrades.

Pods/Logs

The Mission Control Agent needs access to the pod logs to debug issues that may occur.

Replicasets

The Mission Control Agent needs to create and delete pods as needed for each software event broker that it delpoys.

The following Kubernetes YAML descriptor implements the permissions for the service account. In the example below, <target-namespace> is the name of the target namespace in your cluster. You can optionally specify the name of an existing role in your cluster to bind the service account to instead of cloud-agent-role.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: cloud-agent
 namespace: <target-namespace>
imagePullSecrets:
  - name: gcr-reg-secret
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cloud-agent-role-binding
  namespace: <target-namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cloud-agent-role
subjects:
- kind: ServiceAccount
  name: cloud-agent
  namespace: <target-namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Values.serviceAccount.cloudAgent.name }}-role
rules:
  - apiGroups: [""]
    resources: ["secrets", "services", "configmaps"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "update", "patch", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "create", "delete",  "update", "patch", "list", "watch"] 
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups:  ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]