Migrating Machine Image Deployments

This section guides you through the process of migrating an instance of PubSub+ created from a machine image package to a virtual machine (VM) host created in a similar way to that described in Deploying PubSub+ Software Event Brokers in a Production Environment. It describes the process of moving configuration and message delivery state to the new hosts while providing service (in most cases). The process is like the machine image instance replacement upgrade commonly performed with the AWS Machine Image.

Things to consider before starting the procedure:

  • Redundancy must be configured using some stable form of network identity. Updating the redundancy configuration will require a service outage. The best option is to bring the new nodes into the HA group using DNS updates.

  • The original broker instance created from a machine image and the new VM instance that will replace it must be on-line at the same time to transfer the configuration and message state. If the storage containing the broker state can be transferred to the new VM instance by re-attaching storage from the old system to new, this is the simplest method.

At a high level, the following steps are required to migrate from an instance created from a machine image package to a new VM host (assuming the VM host has been configured with storage):

  1. Upgrade the machine image systems to the target version (because upgrading and migrating in a single operation is risky, you should complete the upgrade first).

  2. On the monitor node:

    1. Stop the broker running on the monitor node.

    2. Copy the storage elements to the new VM (or re-attach storage from old to new).

    3. Shut down the machine image node.

    4. Create the new container in the new VM (with the same version as on the original node after upgrade).

    5. Update the DNS so that the other nodes in the HA group will be able to reach the new VM.

    6. Verify that redundancy is back up.

  3. On the standby node:

    1. Stop the broker running on the standby node.

    2. Copy the storage elements to the new VM (or re-attach storage from old to new).

    3. Shut down the machine image node.

    4. Create the new container in the new VM (with the same version as on the original node after upgrade).

    5. Update the DNS so that the other nodes in the HA group will be able to reach the new VM.

    6. Verify that redundancy is back up.

  4. On the active node:

    1. Release activity.

    2. Perform step 3 on the previously active node.

Prerequisites

Before starting the migration, it is assumed that you are familiar with the contents of Deploying PubSub+ Software Event Brokers in a Production Environment. You should have VM instances running their preferred operating system with a container runtime installed (Docker or Podman). You should also have generated a target command-line or systemd unit file to create the new container instance on the VMs. The target command-line does not need to include the redundancy configuration or other configuration keys since the configuration will be migrated from the current broker instances. For a container instance that already has a configuration, for example a configuration that is migrated from a previously running instance, you can use the systemd unit file from Sample Systemd Unit File (with the customizations required for the specific environment). The environment file is not required because the broker instance already has a configuration.

Network and Redundancy

In order to perform the migration in-service, the redundancy configuration cannot change (making changes to the redundancy configuration requires a service outage). Before starting the migration, it is important to confirm that the network identity of the three nodes that make up the HA group is configured in a way that can be transferred to the new hosts during the migration. The easiest way to do this is by a DNS update.

Run the following commands on all three nodes of the HA group to confirm the network identities can be transferred to the new hosts:

solace-primary# show redundancy group
Node Router-Name  Node Type      Address          Status
----------------- -------------- ---------------- ---------
solace-primary*   Message-Router private.solace-1 Online
                                   .test.net
solace-backup     Message-Router private.solace-2 Online
                                   .test.net
solace-monitor    Monitor        private.solace-3 Online
                                   .test.net

* - indicates the current node

If the identities of the nodes in the HA group are configured using DNS names, then you must update these DNS records to bring the new hosts into the HA group. If IP addresses are used, the process is more complicated because both hosts (new and old) must be running at the same time to transfer the configuration (unless the configuration can be transferred without the need to copy). After you transfer the configuration, you need to turn off the existing node and transfer the IP address to the new host. Before starting the migration consider how the new hosts will join the cluster.

Transferring Configuration and Message State

The next thing to consider before starting the migration is how to transfer the configuration and message state from the old node to the new host. There are several ways to do this depending on how the old node was created. With the storage-group (or individual storage elements) on a dedicated device (such as a vSAN volume) you can use hypervisor methods to attach the volume to the new host. If the storage from the old node cannot be attached to the new host, then you will need to copy the data using rsync.

In order to copy the data, you need to first determine the source and target locations. The target location should have been identified while configuring the new host (as described in Deploying PubSub+ Software Event Brokers in a Production Environment). The source of the configuration and message state will depend on how the old node was created. If it was created using a single storage-group, then you can transfer everything in a single command. If it was configured by creating individual storage-elements, then you need to copy the elements one at a time. If there are no special performance requirements for any of the storage elements then you can combine them into a single storage-group as a part of the transfer (see Converting from Multiple Mount Points to a Single Mount Point).

You can find the locations of the storage by logging into the old nodes host (ssh -p 22 sysadmin@host) and executing the following command:

sudo solacectl storage ls

Once you understand the locations of the storage-elements and their new locations for all three nodes, you can begin the migration process.

Procedure

  1. Log into the Solace CLI of each node as an admin user.

  2. Ensure the redundancy Configuration Status is Enabled, and the Redundancy Status is Up on each node. On solace-primary the Message Spool Status should be AD-Active, and on solace-backup the Message Spool Status should be AD-Standby:

    solace-primary# show redundancy
    Configuration Status     : Enabled
    Redundancy Status        : Up
    Operating Mode           : Message Routing Node
    Switchover Mechanism     : Hostlist
    Auto Revert              : No
    Redundancy Mode          : Active/Standby
    Active-Standby Role      : Primary
    Mate Router Name         : solace-backup
    ADB Link To Mate         : Up
    ADB Hello To Mate        : Up
    
    Interface        Static Address           Status
    ---------------- ------------------------ ------
    intf0            2600:1f18:8b2:d275:5ab6: Up
                       c5ff:7a54:567c
    
                                   Primary Virtual Router Backup Virtual Router
                                   ---------------------- ----------------------
    Activity Status                Local Active           Shutdown
    Redundancy Interface Status    Up
    VRRP Status
      intf0                        Initialize
    VRRP Priority                  250
    Message Spool Status           AD-Active
    Priority Reported By Mate      Standby
    solace-backup# show redundancy
    Configuration Status     : Enabled
    Redundancy Status        : Up
    Operating Mode           : Message Routing Node
    Switchover Mechanism     : Hostlist
    Auto Revert              : No
    Redundancy Mode          : Active/Standby
    Active-Standby Role      : Backup
    Mate Router Name         : solace-primary
    ADB Link To Mate         : Up
    ADB Hello To Mate        : Up
    
    Interface        Static Address           Status
    ---------------- ------------------------ ------
    intf0            2600:1f18:8b2:d275:4b64: Up
                       b9bf:2e3f:b0c8
    
                                   Primary Virtual Router Backup Virtual Router
                                   ---------------------- ----------------------
    Activity Status                Shutdown               Mate Active
    Redundancy Interface Status                           Up
    VRRP Status
      intf0                                               Initialize
    VRRP Priority                                         100
    Message Spool Status                                  AD-Standby
    Priority Reported By Mate                             Active
    solace-monitor# show redundancy
    Configuration Status     : Enabled
    Redundancy Status        : Up
    Operating Mode           : Monitoring Node
    Switchover Mechanism     : Hostlist
    Auto Revert              : No
    Redundancy Mode          : Active/Standby
    Active-Standby Role      : None
    Mate Router Name         :
    ADB Link To Mate         : Down
    ADB Hello To Mate        : Down
    Interface        Static Address           Status
    ---------------- ------------------------ ------
    intf0            2600:1f18:8b2:d275:20f6: Up
                       ce76:d651:df78
    
                                    Primary Virtual Router Backup Virtual Router
                                    ---------------------- ----------------------
    Activity Status                 Shutdown               Shutdown
    Redundancy Interface Status
    VRRP Status
      intf0
    VRRP Priority
    Message Spool Status
    Priority Reported By Mate
  3. Log out and log into the machine image host of solace-monitor as sysadmin.

  4. Switch to root user:

    [sysadmin@solace-monitor ~]$ sudo su –
  5. Stop the solace service:

    [root@solace-monitor ~]$ solacectl service stop
  6. Verify that the new VM host is properly configured.

    This includes installing the container runtime:

    For example, on a Red Hat Enterprise Linux 8 host:

    $sudo yum update
    $sudo yum module enable -y container-tools:rhel8
    $sudo yum module install -y container-tools:rhel8

    Configure swap if needed (based on the memory requirements from the System Resource Calculator (described in Deploying PubSub+ Software Event Brokers in a Production Environment).

    Decide where broker’s data will be stored and prepare to transfer it from the host (mounting external devices, creating filesystems, creating directories etc.)

  1. Copy the configuration and state over from the monitor to the new VM.

    On the machine image host, verify the location of the storage elements.

    [root@solace-monitor ~]# solacectl storage ls
    Block Devices:
    Name                                             Size      Note
    xvda                                             8.0G
    └─xvda1                                          8.0G
    
    Storage Volumes:
    Name                                             Size      Used      Available      Path
    /dev/xvda1                                       8.0G      5.2G      2.8G           /
    └─storage-group                                  8.0G      281M      2.8G           /var/lib/docker/volumes/storage-group/_data

    In this case, a single storage-group is configured that contains the data for all of the storage elements. This will be copied over to the new system. From the new system:

    [sysadmin@solace-monitor-new opt]$ sudo rsync -av sysadmin@solace-monitor:/var/lib/docker/volumes/storage-group/_data/* /opt/solace/storage-group

    This command transfers the configuration and message state from the old instance to the new VM (/opt/solace/storage-group or preferred location on the new host).

  2. Update the DNS record to point to the new host.

  3. Create the new container instance.

    In this example, the container will be created using systemd (the target system is running Red Hat Enterprise Linux 8 with the latest container-tools module installed). You can use the sample unit file contained in Deploying PubSub+ Software Event Brokers in a Production Environment to create the solace systemd service.

    The .env file is where a new instance gets its initial configuration. This is not needed in this case since the broker instance is getting its configuration from the machine image instance. The solace.conf file can be modified as required.

    To install and start the new solace systemd service:

    [sysadmin@solace-monitor-new systemd]$ sudo systemctl daemon-reload
    [sysadmin@solace-monitor-new systemd]$ sudo systemctl start solace

    If for some reason the new service does not start properly:

    [sysadmin@solace-monitor-new systemd]$ sudo systemctl status solace
  1. Log in to the Solace CLI on all three nodes and verify that redundancy is Enabled and Up (and that all three nodes are in contact with one and other). If the HA group has been restored to a redundant state, then it is safe to proceed similarly with the standby and then the active node.

  2. Shutdown the old monitor node.

  3. Log into the machine image host of solace-backup as sysadmin.

  4. Switch to root user:

    [sysadmin@solace-backup ~]$ sudo su –
  5. Optional: If you are using MQTT retained messages in your deployment, verify that your retain cache instances are synchronized. For more information, see Verifying Retain Cache Redundancy.

  6. Stop the solace service:

    [root@solace-backup ~]$ solacectl service stop
  7. Configure the new host as in step 6.

  8. Copy the configuration and state over from the old backup node to the new VM.

    On the machine image host, verify the location of the storage elements.

    [root@solace-backup ~]# solacectl storage ls
    Block Devices:
    Name                                              Size      Note
    xvda                                              350.0G
    └─xvda1                                           350.0G
    
    Storage Volumes:
    Name                                              Size      Used      Available      Path
    /dev/xvda1                                        350G      6.6G      344G           /
    └─storage-group                                   350G      1.3G      344G           /var/lib/docker/volumes/storage-group/_data

    Use rsync to transfer the storage elements from the machine image host to the new VM host.

    [sysadmin@solace-backup-new ~]$ sudo rsync -av sysadmin@solace-backup:/var/lib/docker/volumes/storage-group/_data/* /opt/solace/storage-group

    This command transfers the configuration and message state from the old instance to the new VM (/opt/solace/storage-group or preferred location in the new host).

  9. Update the DNS record to point to the new host.

  10. Create the new container instance.

    Create a new container using a system unit file similar to the one used in step 9.

    [sysadmin@solace-backup-new /]$ sudo systemctl daemon-reload
    [sysadmin@solace-backup-new /]$ sudo systemctl start solace
  11. Log in to the Solace CLI of all three broker instances and verify that redundancy is Enabled and Up.

  12. Shut down the old backup node.

  13. In step 20, the redundancy status was verified on all three nodes.

    Fail over to the new backup node by logging into the Solace CLI of the active node as admin user and releasing activity:

    solace-primary> enable
    solace-primary> configure
    solace-primary> redundancy release-activity
    solace-primary(configure/redundancy)# show redundancy
    Configuration Status     : Enabled-Released
    Redundancy Status        : Down
    Operating Mode           : Message Routing Node
    Switchover Mechanism     : Hostlist
    Auto Revert              : No
    Redundancy Mode          : Active/Standby
    Active-Standby Role      : Primary
    Mate Router Name         : solace-backup
    ADB Link To Mate         : Up
    ADB Hello To Mate        : Up
    
    Interface        Static Address           Status
    ---------------- ------------------------ ------
    intf0            2600:1f18:8b2:d275:5ab6: Up
                       c5ff:7a54:567c
    
                                   Primary Virtual Router Backup Virtual Router
                                   ---------------------- ----------------------
    Activity Status                Mate Active            Shutdown
    Redundancy Interface Status    Up
    VRRP Status
    intf0                          Initialize
    VRRP Priority                  0
    Message Spool Status           AD-Standby
    Priority Reported By Mate      Active

    The previously active node now shows the mate as active.

  14. Log in to the new backup node and verify that it has taken activity.

    solace-backup-new# show redundancy
    Configuration Status     : Enabled
    Redundancy Status        : Down
    Operating Mode           : Message Routing Node
    Switchover Mechanism     : Hostlist
    Auto Revert              : No
    Redundancy Mode          : Active/Standby
    Active-Standby Role      : Backup
    Mate Router Name         : solace-primary
    ADB Link To Mate         : Up
    ADB Hello To Mate        : Up
    
    Interface        Static Address           Status
    ---------------- ------------------------ ------
    intf0            2600:1f18:8b2:d275:3ff9: Up
                       76ac:d885:f037
    
                                   Primary Virtual Router Backup Virtual Router
                                   ---------------------- ----------------------
    Activity Status                Shutdown               Local Active
    Redundancy Interface Status                           Up
    VRRP Status
      intf0                                               Initialize
    VRRP Priority                                         240
    Message Spool Status                                  AD-Active
    Priority Reported By Mate                             Release

    The new backup node is now AD-Active.

  15. Log out and log back into the old primary node (now AD-Standby) as sysadmin.

  16. Switch to root user:

    [sysadmin@solace-primary ~]$ sudo su –
  17. Optional: If you're using MQTT retained messages in your deployment, verify that your retain cache instances are synchronized. For more information, see Verifying Retain Cache Redundancy.

  18. Stop the solace service:

    [root@solace-primary ~]$ solacectl service stop
  19. Configure the new host as in step 6.

  20. Copy the configuration and state over from the primary to the new VM.

    On the machine image host, verify the location of the storage elements.

    [root@solace-primary ~]# solacectl storage ls
    Block Devices:
    Name                                              Size      Note
    xvda                                              350.0G
    └─xvda1                                           350.0G
    
    Storage Volumes:
    Name                                              Size      Used      Available 
    Path
    					
    /dev/xvda1                                        350G      6.6G      344G
    /
    
    └─storage-group                                   350G      1.3G      344G 
    /var/lib/docker/volumes/storage-group/_data

    Use rsync to transfer the storage elements from the machine image host to the new VM host.

    [sysadmin@solace-primary-new ~]$ sudo rsync -av sysadmin@solace-primary:/var/lib/docker/volumes/storage-group/_data/* /opt/solace/storage-group

    This command transfers the configuration and message state from the old instance to the new VM (/opt/solace/storage-group or preferred location in the new host).

  21. Update the DNS record to point to the new host.

  22. Create the new container instance.

    Create a new container using a system unit file similar to the one used in step 9.

    [sysadmin@solace-primary-new /]$ sudo systemctl daemon-reload
    [sysadmin@solace-primary-new /]$ sudo systemctl start solace
  23. Shut down the old primary node.

  24. Re-enable redundancy.

    solace-primary-new> enable
    solace-primary-new> configure redundancy
    solace-primary-new> no release-activity
    solace-backup-new> enable
    solace-backup-new> admin
    solace-backup-new> redundancy revert-activity
  25. Verify that redundancy is Up.

    solace-primary-new(configure/redundancy)# show redundancy
    Configuration Status     : Enabled
    Redundancy Status        : Up
    Operating Mode           : Message Routing Node
    Switchover Mechanism     : Hostlist
    Auto Revert              : No
    Redundancy Mode          : Active/Standby
    Active-Standby Role      : Primary
    Mate Router Name         : solace-backup
    ADB Link To Mate         : Up
    ADB Hello To Mate        : Up
    
    Interface        Static Address           Status
    ---------------- ------------------------ ------
    intf0            2600:1f18:8b2:d275:aa94: Up
                       8510:2ec5:883e
    
                                   Primary Virtual Router Backup Virtual Router
                                   ---------------------- ----------------------
    Activity Status                Local Active           Shutdown
    Redundancy Interface Status    Up
    VRRP Status
      intf0                        Initialize
    VRRP Priority                  250
    Message Spool Status           AD-Active
    Priority Reported By Mate      Standby

    Verify that Config-Sync is Up.

    solace-primary-new(configure/redundancy)# show config-sync
    Admin Status                  : Enabled
    Oper Status                   : Up
    SSL Enabled                   : Yes
    Authentication
    Client Certificate
    Maximum Chain Depth           : 3
    Validate Certificate Dates    : Enabled
    Server Certificate Configured : No
    Client-Profile
      TCP
        MSS                       : 1460
        Initial Window            : 2
        Maximum Window            : 256 KB
        Keepalive
          Count                   : 5
          Idle                    : 3 seconds
          Interval                : 1 seconds
    Client
      Name                        : #config-sync/solace-primary
      Connection State            : Connected
      Last Fail Reason            :
    Synchronize
      Username                    : Yes

You have completed this procedure. The system has been migrated.