Migrating Machine Image Deployments
This section guides you through the process of migrating an instance of PubSub+ created from a machine image package to a virtual machine (VM) host created in a similar way to that described in Deploying PubSub+ Software Event Brokers in a Production Environment. It describes the process of moving configuration and message delivery state to the new hosts while providing service (in most cases). The process is like the machine image instance replacement upgrade commonly performed with the AWS Machine Image.
Things to consider before starting the procedure:
-
Redundancy must be configured using some stable form of network identity. Updating the redundancy configuration will require a service outage. The best option is to bring the new nodes into the HA group using DNS updates.
-
The original broker instance created from a machine image and the new VM instance that will replace it must be on-line at the same time to transfer the configuration and message state. If the storage containing the broker state can be transferred to the new VM instance by re-attaching storage from the old system to new, this is the simplest method.
At a high level, the following steps are required to migrate from an instance created from a machine image package to a new VM host (assuming the VM host has been configured with storage):
-
Upgrade the machine image systems to the target version (because upgrading and migrating in a single operation is risky, you should complete the upgrade first).
-
On the monitor node:
-
Stop the broker running on the monitor node.
-
Copy the storage elements to the new VM (or re-attach storage from old to new).
-
Shut down the machine image node.
-
Create the new container in the new VM (with the same version as on the original node after upgrade).
-
Update the DNS so that the other nodes in the HA group will be able to reach the new VM.
-
Verify that redundancy is back up.
-
-
On the standby node:
-
Stop the broker running on the standby node.
-
Copy the storage elements to the new VM (or re-attach storage from old to new).
-
Shut down the machine image node.
-
Create the new container in the new VM (with the same version as on the original node after upgrade).
-
Update the DNS so that the other nodes in the HA group will be able to reach the new VM.
-
Verify that redundancy is back up.
-
-
On the active node:
-
Release activity.
-
Perform step 3 on the previously active node.
-
Prerequisites
Before starting the migration, it is assumed that you are familiar with the contents of Deploying PubSub+ Software Event Brokers in a Production Environment. You should have VM instances running their preferred operating system with a container runtime installed (Docker or Podman). You should also have generated a target command-line or systemd unit file to create the new container instance on the VMs. The target command-line does not need to include the redundancy configuration or other configuration keys since the configuration will be migrated from the current broker instances. For a container instance that already has a configuration, for example a configuration that is migrated from a previously running instance, you can use the systemd unit file from Sample Systemd Unit File (with the customizations required for the specific environment). The environment file is not required because the broker instance already has a configuration.
Network and Redundancy
In order to perform the migration in-service, the redundancy configuration cannot change (making changes to the redundancy configuration requires a service outage). Before starting the migration, it is important to confirm that the network identity of the three nodes that make up the HA group is configured in a way that can be transferred to the new hosts during the migration. The easiest way to do this is by a DNS update.
Run the following commands on all three nodes of the HA group to confirm the network identities can be transferred to the new hosts:
solace-primary# show redundancy group
Node Router-Name Node Type Address Status
----------------- -------------- ---------------- ---------
solace-primary* Message-Router private.solace-1 Online
.test.net
solace-backup Message-Router private.solace-2 Online
.test.net
solace-monitor Monitor private.solace-3 Online
.test.net
* - indicates the current node
If the identities of the nodes in the HA group are configured using DNS names, then you must update these DNS records to bring the new hosts into the HA group. If IP addresses are used, the process is more complicated because both hosts (new and old) must be running at the same time to transfer the configuration (unless the configuration can be transferred without the need to copy). After you transfer the configuration, you need to turn off the existing node and transfer the IP address to the new host. Before starting the migration consider how the new hosts will join the cluster.
Transferring Configuration and Message State
The next thing to consider before starting the migration is how to transfer the configuration and message state from the old node to the new host. There are several ways to do this depending on how the old node was created. With the storage-group (or individual storage elements) on a dedicated device (such as a vSAN volume) you can use hypervisor methods to attach the volume to the new host. If the storage from the old node cannot be attached to the new host, then you will need to copy the data using rsync.
In order to copy the data, you need to first determine the source and target locations. The target location should have been identified while configuring the new host (as described in Deploying PubSub+ Software Event Brokers in a Production Environment). The source of the configuration and message state will depend on how the old node was created. If it was created using a single storage-group, then you can transfer everything in a single command. If it was configured by creating individual storage-elements, then you need to copy the elements one at a time. If there are no special performance requirements for any of the storage elements then you can combine them into a single storage-group as a part of the transfer (see Converting from Multiple Mount Points to a Single Mount Point).
You can find the locations of the storage by logging into the old nodes host (ssh -p 22 sysadmin@host) and executing the following command:
sudo solacectl storage ls
Once you understand the locations of the storage-elements and their new locations for all three nodes, you can begin the migration process.
Procedure
-
Log into the Solace CLI of each node as an admin user.
-
Ensure the redundancy
Configuration StatusisEnabled, and theRedundancy StatusisUpon each node. Onsolace-primarytheMessage Spool Statusshould beAD-Active, and onsolace-backuptheMessage Spool Statusshould beAD-Standby:solace-primary# show redundancy Configuration Status : Enabled Redundancy Status : Up Operating Mode : Message Routing Node Switchover Mechanism : Hostlist Auto Revert : No Redundancy Mode : Active/Standby Active-Standby Role : Primary Mate Router Name : solace-backup ADB Link To Mate : Up ADB Hello To Mate : Up Interface Static Address Status ---------------- ------------------------ ------ intf0 2600:1f18:8b2:d275:5ab6: Up c5ff:7a54:567c Primary Virtual Router Backup Virtual Router ---------------------- ---------------------- Activity Status Local Active Shutdown Redundancy Interface Status Up VRRP Status intf0 Initialize VRRP Priority 250 Message Spool Status AD-Active Priority Reported By Mate Standbysolace-backup# show redundancy Configuration Status : Enabled Redundancy Status : Up Operating Mode : Message Routing Node Switchover Mechanism : Hostlist Auto Revert : No Redundancy Mode : Active/Standby Active-Standby Role : Backup Mate Router Name : solace-primary ADB Link To Mate : Up ADB Hello To Mate : Up Interface Static Address Status ---------------- ------------------------ ------ intf0 2600:1f18:8b2:d275:4b64: Up b9bf:2e3f:b0c8 Primary Virtual Router Backup Virtual Router ---------------------- ---------------------- Activity Status Shutdown Mate Active Redundancy Interface Status Up VRRP Status intf0 Initialize VRRP Priority 100 Message Spool Status AD-Standby Priority Reported By Mate Activesolace-monitor# show redundancy Configuration Status : Enabled Redundancy Status : Up Operating Mode : Monitoring Node Switchover Mechanism : Hostlist Auto Revert : No Redundancy Mode : Active/Standby Active-Standby Role : None Mate Router Name : ADB Link To Mate : Down ADB Hello To Mate : Down Interface Static Address Status ---------------- ------------------------ ------ intf0 2600:1f18:8b2:d275:20f6: Up ce76:d651:df78 Primary Virtual Router Backup Virtual Router ---------------------- ---------------------- Activity Status Shutdown Shutdown Redundancy Interface Status VRRP Status intf0 VRRP Priority Message Spool Status Priority Reported By Mate -
Log out and log into the machine image host of
solace-monitorassysadmin. -
Switch to root user:
[sysadmin@solace-monitor ~]$ sudo su –
-
Stop the solace service:
[root@solace-monitor ~]$ solacectl service stop
-
Verify that the new VM host is properly configured.
This includes installing the container runtime:
For example, on a Red Hat Enterprise Linux 8 host:
$sudo yum update $sudo yum module enable -y container-tools:rhel8 $sudo yum module install -y container-tools:rhel8
Configure swap if needed (based on the memory requirements from the System Resource Calculator (described in Deploying PubSub+ Software Event Brokers in a Production Environment).
Decide where broker’s data will be stored and prepare to transfer it from the host (mounting external devices, creating filesystems, creating directories etc.)
-
Copy the configuration and state over from the monitor to the new VM.
On the machine image host, verify the location of the storage elements.
[root@solace-monitor ~]# solacectl storage ls Block Devices: Name Size Note xvda 8.0G └─xvda1 8.0G Storage Volumes: Name Size Used Available Path /dev/xvda1 8.0G 5.2G 2.8G / └─storage-group 8.0G 281M 2.8G /var/lib/docker/volumes/storage-group/_data
In this case, a single storage-group is configured that contains the data for all of the storage elements. This will be copied over to the new system. From the new system:
[sysadmin@solace-monitor-new opt]$ sudo rsync -av sysadmin@solace-monitor:/var/lib/docker/volumes/storage-group/_data/* /opt/solace/storage-group
This command transfers the configuration and message state from the old instance to the new VM (
/opt/solace/storage-groupor preferred location on the new host). -
Update the DNS record to point to the new host.
-
Create the new container instance.
In this example, the container will be created using systemd (the target system is running Red Hat Enterprise Linux 8 with the latest container-tools module installed). You can use the sample unit file contained in Deploying PubSub+ Software Event Brokers in a Production Environment to create the solace systemd service.
The
.envfile is where a new instance gets its initial configuration. This is not needed in this case since the broker instance is getting its configuration from the machine image instance. Thesolace.conffile can be modified as required.To install and start the new solace systemd service:
[sysadmin@solace-monitor-new systemd]$ sudo systemctl daemon-reload [sysadmin@solace-monitor-new systemd]$ sudo systemctl start solace
If for some reason the new service does not start properly:
[sysadmin@solace-monitor-new systemd]$ sudo systemctl status solace
-
Log in to the Solace CLI on all three nodes and verify that redundancy is
EnabledandUp(and that all three nodes are in contact with one and other). If the HA group has been restored to a redundant state, then it is safe to proceed similarly with the standby and then the active node. -
Shutdown the old monitor node.
-
Log into the machine image host of
solace-backupassysadmin. -
Switch to root user:
[sysadmin@solace-backup ~]$ sudo su –
-
Optional: If you are using MQTT retained messages in your deployment, verify that your retain cache instances are synchronized. For more information, see Verifying Retain Cache Redundancy.
-
Stop the solace service:
[root@solace-backup ~]$ solacectl service stop
-
Configure the new host as in step 6.
-
Copy the configuration and state over from the old backup node to the new VM.
On the machine image host, verify the location of the storage elements.
[root@solace-backup ~]# solacectl storage ls Block Devices: Name Size Note xvda 350.0G └─xvda1 350.0G Storage Volumes: Name Size Used Available Path /dev/xvda1 350G 6.6G 344G / └─storage-group 350G 1.3G 344G /var/lib/docker/volumes/storage-group/_data
Use rsync to transfer the storage elements from the machine image host to the new VM host.
[sysadmin@solace-backup-new ~]$ sudo rsync -av sysadmin@solace-backup:/var/lib/docker/volumes/storage-group/_data/* /opt/solace/storage-group
This command transfers the configuration and message state from the old instance to the new VM (
/opt/solace/storage-groupor preferred location in the new host). -
Update the DNS record to point to the new host.
-
Create the new container instance.
Create a new container using a system unit file similar to the one used in step 9.
[sysadmin@solace-backup-new /]$ sudo systemctl daemon-reload [sysadmin@solace-backup-new /]$ sudo systemctl start solace
-
Log in to the Solace CLI of all three broker instances and verify that redundancy is
EnabledandUp. -
Shut down the old backup node.
-
In step 20, the redundancy status was verified on all three nodes.
Fail over to the new backup node by logging into the Solace CLI of the active node as admin user and releasing activity:
solace-primary> enable solace-primary> configure solace-primary> redundancy release-activity
solace-primary(configure/redundancy)# show redundancy Configuration Status : Enabled-Released Redundancy Status : Down Operating Mode : Message Routing Node Switchover Mechanism : Hostlist Auto Revert : No Redundancy Mode : Active/Standby Active-Standby Role : Primary Mate Router Name : solace-backup ADB Link To Mate : Up ADB Hello To Mate : Up Interface Static Address Status ---------------- ------------------------ ------ intf0 2600:1f18:8b2:d275:5ab6: Up c5ff:7a54:567c Primary Virtual Router Backup Virtual Router ---------------------- ---------------------- Activity Status Mate Active Shutdown Redundancy Interface Status Up VRRP Status intf0 Initialize VRRP Priority 0 Message Spool Status AD-Standby Priority Reported By Mate ActiveThe previously active node now shows the mate as active.
-
Log in to the new backup node and verify that it has taken activity.
solace-backup-new# show redundancy Configuration Status : Enabled Redundancy Status : Down Operating Mode : Message Routing Node Switchover Mechanism : Hostlist Auto Revert : No Redundancy Mode : Active/Standby Active-Standby Role : Backup Mate Router Name : solace-primary ADB Link To Mate : Up ADB Hello To Mate : Up Interface Static Address Status ---------------- ------------------------ ------ intf0 2600:1f18:8b2:d275:3ff9: Up 76ac:d885:f037 Primary Virtual Router Backup Virtual Router ---------------------- ---------------------- Activity Status Shutdown Local Active Redundancy Interface Status Up VRRP Status intf0 Initialize VRRP Priority 240 Message Spool Status AD-Active Priority Reported By Mate ReleaseThe new backup node is now
AD-Active. -
Log out and log back into the old primary node (now
AD-Standby) assysadmin. -
Switch to root user:
[sysadmin@solace-primary ~]$ sudo su –
-
Optional: If you're using MQTT retained messages in your deployment, verify that your retain cache instances are synchronized. For more information, see Verifying Retain Cache Redundancy.
-
Stop the solace service:
[root@solace-primary ~]$ solacectl service stop
-
Configure the new host as in step 6.
-
Copy the configuration and state over from the primary to the new VM.
On the machine image host, verify the location of the storage elements.
[root@solace-primary ~]# solacectl storage ls Block Devices: Name Size Note xvda 350.0G └─xvda1 350.0G Storage Volumes: Name Size Used Available Path /dev/xvda1 350G 6.6G 344G / └─storage-group 350G 1.3G 344G /var/lib/docker/volumes/storage-group/_data
Use rsync to transfer the storage elements from the machine image host to the new VM host.
[sysadmin@solace-primary-new ~]$ sudo rsync -av sysadmin@solace-primary:/var/lib/docker/volumes/storage-group/_data/* /opt/solace/storage-group
This command transfers the configuration and message state from the old instance to the new VM (
/opt/solace/storage-groupor preferred location in the new host). -
Update the DNS record to point to the new host.
-
Create the new container instance.
Create a new container using a system unit file similar to the one used in step 9.
[sysadmin@solace-primary-new /]$ sudo systemctl daemon-reload [sysadmin@solace-primary-new /]$ sudo systemctl start solace
-
Shut down the old primary node.
-
Re-enable redundancy.
solace-primary-new> enable solace-primary-new> configure redundancy solace-primary-new> no release-activity
solace-backup-new> enable solace-backup-new> admin solace-backup-new> redundancy revert-activity
-
Verify that redundancy is
Up.solace-primary-new(configure/redundancy)# show redundancy Configuration Status : Enabled Redundancy Status : Up Operating Mode : Message Routing Node Switchover Mechanism : Hostlist Auto Revert : No Redundancy Mode : Active/Standby Active-Standby Role : Primary Mate Router Name : solace-backup ADB Link To Mate : Up ADB Hello To Mate : Up Interface Static Address Status ---------------- ------------------------ ------ intf0 2600:1f18:8b2:d275:aa94: Up 8510:2ec5:883e Primary Virtual Router Backup Virtual Router ---------------------- ---------------------- Activity Status Local Active Shutdown Redundancy Interface Status Up VRRP Status intf0 Initialize VRRP Priority 250 Message Spool Status AD-Active Priority Reported By Mate StandbyVerify that Config-Sync is
Up.solace-primary-new(configure/redundancy)# show config-sync Admin Status : Enabled Oper Status : Up SSL Enabled : Yes Authentication Client Certificate Maximum Chain Depth : 3 Validate Certificate Dates : Enabled Server Certificate Configured : No Client-Profile TCP MSS : 1460 Initial Window : 2 Maximum Window : 256 KB Keepalive Count : 5 Idle : 3 seconds Interval : 1 seconds Client Name : #config-sync/solace-primary Connection State : Connected Last Fail Reason : Synchronize Username : Yes
You have completed this procedure. The system has been migrated.