Mission Control Agent

The Mission Control Agent is an important microservice that is required to deploy and manage PubSub+ Event Broker: Cloud (event broker services). The high-level actions (non-granular calls) that the Mission Control Agent makes are limited to the following functions:

Manages the life cycle and configuration of event broker services in the deployment that includes the creation, deletion, updates, and configuration options on all event broker services in the data center
Configures Datadog agents. A Datadog agent is part of the deployment that send metrics and monitoring data from an event broker service to the central monitoring service. This is required for an event broker services to be monitored and necessary for PubSub+ Cloud to function. For more information about the central monitoring service and Datadog agents, see Central Monitoring Service and Datadog Agents.
Send system-level diagnostic information of the event broker services to the PubSub+ Home Cloud to allow for rapid response to address any issues. Information provided is solely for to help with the recovery of event broker services to save recovery and costs.
Transfer logs and monitoring traffic to the Home Cloud. No details of the network or personal information is sent.

Depending on deployment configuration, the Mission Control Agent typically resides in the same network as where the event broker services are deployed. In the following architecture diagram, the red box highlights the Mission Control Agent.

The Mission Control Agent communication with the PubSub+ Home Cloud is summarized as follows:

the Mission Control Agent always initiates the connection to the Home Cloud. This is a one-way initiation - the PubSub+ Home Cloud never initiates the call.
the Mission Control Agent never transmits any messaging data or personal account information
the Mission Control Agent receives and sends only information related to orchestrating event broker services (includes the creation, deletion, and configuration of event broker services) and monitoring system-level health checks
the Mission Control Agent authenticates with the Home Cloud using an API token, allowing it to connect securely to the Home Cloud's internal event broker to exchange information; the Mission Control Agent does not require a public IP address

Since the Mission Control Agent is a microservice, it does require a minimal number of permissions to run within the customer's network to manage the life cycle and configuration of event broker services.

For details, see the sections that follow:

Mission Control Agent Installation
Permissions Required for the Mission Control Agent
Information Exchanged Between the PubSub+ Home Cloud and the Mission Control Agent
Information Exchanged Between the Mission Control Agent and the Centralized Monitoring Service
Mission Control Agent Communication within the Kubernetes Cluster
Mission Control Agent Connectivity

Mission Control Agent Installation

During installation, the Mission Control Agent is bootstrapped with the configuration it needs to manage event broker services in the customer's environment. This bootstrapping information is never shared. This configuration includes:

for private cloud deployments:
- restricted IaaS credentials for the customer's virtual network or VPC
- IaaS infrastructure details, such as the names of the virtual network or VPC and subnets
- SSH Key Pair (.pem file) used to configure cloud instances or virtual machines for event broker services
for Kubernetes deployments:
- the data center ID and target namespace
- restricted service account credentials. This service-account is bound to the admin role, which is scoped to the target namespace.
- infrastructure details, such as the names of the primary, backup, and monitoring zones, configuration details for Datadog, and storage and load balancer parameters

Additionally, the customer data center is registered with the Home Cloud and is locked to the customer account. All Mission Control Agents running in the customer virtual network or Kubernetes cluster are authenticated and similarly locked to the customer's PubSub+ Cloud account using API keys.

Permissions Required for the Mission Control Agent

The Mission Control Agent requires a service account with the specific permissions. The service account requires a minimal set of permissions that permit the Mission Control Agent to manage and configure event broker services and communicate with the PubSub+ Home Cloud.

Depending on the deployment chosen, the following permissions are required for the Mission Control Agent:

Kubernetes

The permissions listed below apply to all Kubernetes-based deployments that include Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE).

The Mission Control Agent is assigned a service account called cloud-agent; this account is created automatically by the Helm chart.

This service account is bound to a role called cloud-agent-role, which is scoped to the target namespace. Solace does not support the integration of event broker services with service meshes, such as Itsio, Cillium, and Linkerd. If your cluster has a service mesh, this namespace must be excluded from it. The service account is also bound to the Docker Registry secret which gives it access to Solace's enterprise Docker images.

The cloud-agent-role gives the Mission Control Agent permissions for the following namespace resources:

Secrets: The Mission Control Agent needs to create, update, and delete secrets for the event broker service it manages.
Services: The Mission Control Agent needs to create, update, and delete services to expose theevent broker serviceTCP ports to its clients.
configmaps: The Mission Control Agent needs to create, update, and delete configmaps for the event broker service it manages.
Pods: The Mission Control Agent needs to update and delete pods for the event broker service it manages.
Pods/Exec: The Mission Control Agent needs to execute commands in the event broker service's pods for certain operations such as in-service upgrades and configuring the monitoring agent.
Persistent Volume Claims: The Mission Control Agent needs to update and delete PVCs for the event broker service it manages.
Events: The Mission Control Agent needs to retrieve Events generated by Statefulsets, Jobs, and Services to report scheduling errors and Service creation failures.
Statefulsets: The Mission Control Agent uses Statefulsets as controllers for the event broker service pods. It needs to create, update and delete Statefulsets as part of managing the lifecycle of the event broker services.
Deployments: The Mission Control Agent needs deployment permissions to perform self-upgrades and to create, upgrade, and delete distributed tracing deployments.
Jobs: The Mission Control Agent needs to create, monitor, and delete Jobs to perform schema migration during upscaling operations. This is accomplished by launching a Pod via the Job controller.
Pod Disruption Budgets: The Mission Control Agent creates a Pod Disruption Budget (PDB) for each software event broker that it deploys. It also manages the PDBs afterward.
PDBs are required by Kubernetes worker node upgrades to ensure that event broker services remain operational during Kubernetes rolling upgrades.
Pods/Logs: The Mission Control Agent needs access to the pod logs to debug issues that may occur.
Replicasets: The Mission Control Agent needs to create and delete pods as needed for each software event broker that it delpoys.

The following Kubernetes YAML descriptor implements the permissions for the service account. In the example below, <target-namespace> is the name of the target namespace in your cluster. You can optionally specify the name of an existing role in your cluster to bind the service account to instead of cloud-agent-role.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: cloud-agent
 namespace: <target-namespace>
imagePullSecrets:
  - name: gcr-reg-secret
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cloud-agent-role-binding
  namespace: <target-namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cloud-agent-role
subjects:
- kind: ServiceAccount
  name: cloud-agent
  namespace: <target-namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Values.serviceAccount.cloudAgent.name }}-role
rules:
  - apiGroups: [""]
    resources: ["secrets", "services", "configmaps"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "update", "patch", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "create", "delete",  "update", "patch", "list", "watch"] 
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups:  ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["create", "get", "update", "patch", "delete", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]

Amazon Web Services (AWS)—VM-Based Deployments Only

Support for VM-based deployments is now deprecated and version 10.0.1 was the last event broker release that supported deployments in VM-based regions. For more details, see the Deprecated Features list.

The permissions below apply to VM-based deployments. For permissions for Amazon Elastic Kubernetes Service (EKS), see the required Kubernetes permissions.

The Mission Control Agent is assigned an AWS account with API access that's required to have sufficient permissions to install a datacenter and event broker services. Also, the user’s access key and secret access key are required.

The following permissions are required:

EC2.*.

This permission allows PubSub+ Cloud to create and manage EC2 instances for both the Mission Control Agent and event brokers, network interfaces (including attachments), and the storage.

Action

ec2:*
iam:CreateServiceLinkedRole

Action (Deny):

ec2:AcceptVpcPeeringConnection
ec2:AssociateClientVpnTargetNetwork
ec2:CreateVPC
ec2:DeleteVPC
ec2:CreateVPNConnection
ec2:DeleteRoute

Resource:

Network Elastic Load Balancing (ELB)

This permission allows PubSub+ Cloud to create a network load balancers that front the Event broker services.

Action

elasticloadbalancing:*

Resource:

S3

This permission allows the resources (Mission Control Agent) to create an S3 bucket and save a diagnostic package from the event broker service to the S3 bucket.

The diagnostic package contains logs files, information about the host instance, and any generated core files that Solace can download from the S3 bucket using a time-limited, signed URL. The diagnostic package creation is customer-initiated.

This permission is temporarily required when you need to capture diagnostic information and share the information with Solace.

Action

s3:*

Resource:

arn:aws:s3:::solace-diag-*
arn:aws:s3:::solace-diag-*/*

CloudFormation

This permission allows PubSub+ Cloud to create a CloudFormation stack for the data center infrastructure and the event broker services. This permission is also used for the installation of the Mission Control Agent.

Action:

cloudformation:*

Resource:

Azure—VM-Based Deployments Only

The permissions below apply to VM-based deployments in Azure. For permissions for Azure Kubernetes Service (AKS), see the required Kubernetes permissions.

The Mission Control Agent is assigned a service account called cloud-agent and this account is created via an install script that Solace provides with the following permissions:

{
  "Name": "cloud-agent",
  "IsCustom": "true",
  "Description": "Mission Control Agent",
  "Actions": [
    "Microsoft.Authorization/locks/*",
    "Microsoft.Resources/deployments/*",
    "Microsoft.Resources/subscriptions/resourceGroups/*",
    "Microsoft.Compute/images/*",
    "Microsoft.Compute/snapshots/*",
    "Microsoft.Compute/disks/*",
    "Microsoft.Compute/virtualMachines/*",
    "Microsoft.Compute/availabilitySets/*",
    "Microsoft.Insights/Metrics/*",
    "Microsoft.Insights/MetricDefinitions/*",
    "Microsoft.Network/networkInterfaces/*",
    "Microsoft.Network/loadBalancers/*",
    "Microsoft.Network/networkSecurityGroups/*",
    "Microsoft.Network/locations/usages/read",
    "Microsoft.Network/virtualNetworks/*",
    "Microsoft.Network/publicIPAddresses/*",
    "Microsoft.Storage/storageAccounts/*"
  ],
  "NotActions": [],
  "DataActions": [],
  "NotDataActions": [],
  "AssignableScopes": ["/subscriptions/<subscriptionId>"]
}

Information Exchanged Between the PubSub+ Home Cloud and the Mission Control Agent

The Mission Control Agent authenticates with the PubSub+ Home Cloud using an API token, allowing it to connect securely to the Home Cloud to exchange information. The Mission Control Agent always initiates of the connection with the Home Cloud—in other words, the initiation of the connection is unidirectional. Once connected, the Mission Control Agent information is exchanged.

The Mission Control Agent never transmits messaging data or any personal account information—only control plane (management) information. The Mission Control Agent receives management data that consist of configuration information for the event broker services and shares only metadata related to orchestrating an event broker service and monitoring data related to system-level checks on the event broker service components.

The Mission Control Agent never shares any customer-provided bootstrapping configuration with the Home Cloud. The Home Cloud and the Mission Control Agent do not exchange any access keys or credentials information of any user. The Mission Control Agent authenticates with the Home Cloud using an API token.

The data exchanged between the Mission Control Agent consists of information collected only from the event broker services that the Mission Control Agent has permission to access.

The information that is exchanged between the Home Cloud and the Mission Control Agent is as follows:

Event broker services — The metadata required to configure an event broker service, including cloud instance, IaaS configuration details, configuration specifications (e.g., ARM templates, CloudFormation stacks, Helm charts), and upgrade the Mission Control Agent. This includes responses codes and status information for Home Cloud initiated actions (upgrades, service creation and deletion, etc.), confirmation as to whether the action completed as intended are collected.
Heart beats — Health checks for various components of the event broker services are logged.
Response codes and status — The Home Cloud initiated actions (upgrades, service creation and deletion, etc.), confirmation as to whether the action completed as intended are collected.
Non-grandular commands — Only high-level commands to create, configure, and manage event broker services are exchanged. There are never any granular commands ever sent from the Home Cloud to the Mission Control Agent.
DNS configuration — The Mission Control Agent coordinates DNS configuration with the Home Cloud. This information is necessary to communicate with the event broker services.
Certificates — The Home Cloud provides the signed URL to the Mission Control Agent to allow the Mission Control Agent to load the certificates to the event broker services.
Diagnostic information — The Mission Control Agent collects and shares diagnostic information to assist with incident handling. No personal information is collected as part of the diagnostic information.

Information Exchanged Between the Mission Control Agent and the Centralized Monitoring Service

Solace uses the Datadog cloud application for its central monitoring service component. The Datadog agent is the monitoring component that resides in a deployment. There is one Datadog agent per event broker service. For more details, see the deployment architecture diagrams for your deployment. For example, see PubSub+ Cloud can be installed in a customer's on-premises or cloud-based Kubernetes cluster. for a Kubernetes-based deployment.

The Mission Control Agent configures the Datadog agents for monitoring, but no other information is exchanged. The Datadog agents collect and send monitoring information (logs and metrics) to the central monitoring service about an event broker service and metrics about the Mission Control Agent. For more details about the central monitoring service, see Central Monitoring Service and Datadog Agents.

Mission Control Agent Communication within the Kubernetes Cluster

The Mission Control Agent communicates with various Kubernetes cluster components, which includes the event broker services deployed in the cluster.

The communication with the event broker services consists of information exchanged to manage their lifecycle and configuration operations that includes creating, deleting, updating, and configuring event broker services. This communication uses a combination of SEMPv1, SEMPv2, and Solace CLI commands (via Kubernetes exec mode).
Communication within the Kubernetes cluster is for Kubernetes API and kubelets.

The following table shows the destination ports that are used within the Kubernetes cluster (source ports are ephemeral):

Port Requirement Within Kubernetes Cluster

1943

Port	Requirement Within Kubernetes Cluster
1943	This port is required for communication between the event broker services and the Mission Control Agent to manage lifecycle operations. External SEMP requests are that are received on port 943 are forwarded to 1943 to the containers for the event broker services.
443	This standard port is required for interaction between the Mission Control Agent and the Kubernetes API that is required for control plane operations to manage event broker services.
6443	This standard port is required for communication of between the Mission Control Agent and kubelets (including `exec kubectl`) required for control plane operations to manage event broker services.

This port is required for communication between the event broker services and the Mission Control Agent to manage lifecycle operations.

External SEMP requests are that are received on port 943 are forwarded to 1943 to the containers for the event broker services.

443 This standard port is required for interaction between the Mission Control Agent and the Kubernetes API that is required for control plane operations to manage event broker services.

6443

This standard port is required for communication of between the Mission Control Agent and kubelets (including exec kubectl) required for control plane operations to manage event broker services.

Port 1943 must be enabled within the Kubernetes cluster, however Port 1943 does not need to be enabled for incoming connections to the Kubernetes cluster for event broker services.

Mission Control Agent Connectivity

Connectivity for the Mission Control Agent does not require a public IP address and the Mission Control Agent always initiates the connection to the PubSub+ Home Cloud. The Mission Control Agent communicates with the Home Cloud in the following ways:

securely over port 55443 using the Solace Message Format (SMF) protocol over TCP
Securely over HTTP port 443 during installation and bootstrapping. This is also how the docker image for the Mission Control Agent is retrieved from the Solace registry for Kubernetes installations

For Kubernetes deployments, Docker images are part of a deployment. The Docker images that are used in deployment are available via Kubernetes to the Google Container Registry (gcr.io) through port 443 (secure port). Having the Docker images available in the gcr.io is ideal if your security policies require that all images are scanned prior to deployment.

For VM-based deployments on AWS and Azure, security updates are delivered on port 80 for the Mission Control Agent's EC2/VM image. If your security policies don't permit port 80 to be accessible, the Mission Control Agent EC2 (for AWS) or VM (for Azure) must be periodically recreated with an updated AMI/VM to obtain the latest security patches. Contact Solace to obtain an updated AMI/VM.

Provide feedback