Replacing a LUN and Migrating the Disk Spool Files

The procedures discussed in this section are applicable only to Solace PubSub+ appliances.

This section describes how to replace a LUN for a redundant Solace PubSub+ appliance pair, and then migrate the disk spool files from the old LUN to the new LUN without losing spooled Guaranteed messages. Replacing a LUN is often done to provide a larger LUN to so that the message-spool size can be increased.

If you require further assistance, or have any questions regarding this procedure, contact Solace.

Before You Begin

This procedure requires the adkey-tool script, which is included only with release 9.8.0 or later. If you are using an older release, please contact Solace to obtain this script.

After you have obtained the script, you must copy it to your primary appliance and then test that it is available before beginning either of the procedures below.

To test that the adkey-tool script is available on the appliance, do the following:

On the primary appliance, enter the following command to elevate to the support user, and then enter the support user's password when prompted.
```
solace-primary# shell redundantLunMigration
login: support
Password:
```
Enter the following command to elevate to the root user, and then enter the root user's password when prompted:
```
[support@solace-primary ~]$ su -
Password:
```

Execute the adkey-tool --help command:

[root@solace-primary ~]# adkey-tool -h

If the expected adkey-tool output does not appear, do not proceed with either of the procedures below. Contact Solace for assistance.

Return to the CLI:

[root@solace-primary ~]# exit
[support@solace-primary ~]$ exit

Procedure

Ensure that you have read and followed the instructions in Before You Begin.

This procedure disables service on both appliances in the redundant pair.
To prevent any loss of configuration, do not make any additional unrelated configuration changes to either of the appliances in the redundant pair while performing this procedure.

We recommend that the new LUN be the same size or larger than the old LUN. If the new LUN is smaller, ensure that there is sufficient space available for all files to be copied from the old LUN—the files must be copied successfully for this procedure to succeed.
To replace a LUN, first ensure that the current system still works with the old LUN, then check to see if the new LUN is provisioned. Both LUNs must be present in the system before migrating the disk spool files to the new LUN. For details, refer to Configuring an External Disk Array for Guaranteed Messaging.
The GDisk utility is used to create and modify partitions on a LUN. New partitions use an ext4 file system.
Operations listed below that state they can only be performed by root can also be performed by a Sysadmin User. For information on configuring Sysadmin Users, refer to Configuring Multiple Linux Shell Users.

To replace a LUN for a redundant appliance pair and migrate the disk spool files, perform the following steps:

Ensure that both the primary and backup appliances are in the correct state:
1. On the primary appliance, run the show message-spool detail command and verify the following:
  - Config Status is Enabled (Primary).
  - Operational status is AD-Active.
  - Valid keys are displayed for both primary and backup disks. The particular key values and the relationship between them are not important, as long as valid values are shown for both primary and backup keys.
  For example:
```
solace-primary> show message-spool detail
Config Status:                                    Enabled (Primary)
 
. . .

Operational Status:                               AD-Active

. . . 
                            
Disk Contents:                 Ready
  Disk Key (Primary):            192.168.164.180:8,3
  Disk Key (Backup):             192.168.164.180:7,3
```
  If the primary appliance does not show a status of AD-Active OR does not show valid keys for both disks, follow the steps below. Be aware this will cause a service disruption as activity fails over to the backup, and again as activity is reverted to the primary.
  1. On the primary appliance, run the release-activity Redundancy CONFIG command to fail over to the backup appliance. Refer to Releasing Event Broker Activity for instructions.
  2. On the primary appliance, run the no release-activity Redundancy CONFIG to resume service. Refer to Releasing Event Broker Activity for instructions.
  3. On the backup appliance, run the redundancy revert-activity Admin EXEC command to release activity. See Forcing Backups to Give Up Activity to Primaries for instructions.
  4. Restart this procedure at Step 1 to ensure that the primary appliance is now in the correct state.
2. On the backup appliance, run the show message-spool command and verify the following:
  - Config Status is Enabled (Backup).
  - Operational Status is AD-Standby.
  For example:
```
solace-backup> show message-spool
Config Status:                                    Enabled (Backup)
 
. . .

Operational Status:                               AD-Standby
```

Identify the WWN of the old LUN.

Enter the following command on the primary appliance.

solace-primary> show message-spool
Config Status:                                    Enabled (Primary)
Maximum Spool Usage:                              10000 MB
Spool While Charging:                             No
Spool Without Flash Card:                         No
Using Internal Disk:                              No
Disk Array WWN:                                   60:06:01:60:4d:30:1c:00:8e:29:1b:b6:a6:d6:e8:11

. . .

The examples in this procedure use 60:06:01:60:4d:30:1c:00:8e:29:1b:b6:a6:d6:e8:11 as the WWN of the old LUN.

To detect the new LUN on the primary appliance, perform the following steps:

Enter the following command to elevate to the support user, and then enter the support user's password when prompted.
```
solace-primary# shell redundantLunMigration
login: support
Password:
```
Enter the following command to elevate to the root user, and then enter the root user's password when prompted:
```
[support@solace-primary ~]$ su -
Password:
```

Check that the current LUN is visible.

[root@solace-primary ~]# multipath -ll
3600601604d301c008e291bb6a6d6e811 dm-0 DGC     ,RAID 0 size=300G features='2 queue_if_no_path

. . .

[root@solace-primary ~]#

Rescan the SCSI bus to add the new LUN.

[root@solace-primary ~]# rescan-scsi-bus.sh --nosync -f -r -m
[root@solace-primary ~]# rescan-scsi-bus.sh -a

If the previous step failed, enter these commands:
```
[root@solace-primary ~]# rescan-scsi-bus.sh -i -a
[root@solace-primary ~]# rescan-scsi-bus.sh --nosync -f -r -m
[root@solace-primary ~]# rescan-scsi-bus.sh -a
```
The -i option causes the link to all external disks to go down, and affects PubSub+ Cache if it is running on this node.
Check that the new LUN has been added.
```
[root@solace-primary ~]# multipath -ll
3600601604d301c008e291bb6a6d6e811 dm-0 DGC     ,RAID 0 size=300G features='2 queue_if_no_path

. . .

360014057d24f4b77681435faf684d587 dm-3 LIO-ORG ,sdc1-4 size=800G features='1 queue_if_no_path' hwhandler='0' wp=rw

. . .

[root@solace-primary ~]#
```
The examples in this procedure use 60:01:40:57:d2:4f:4b:77:68:14:35:fa:f6:84:d5:87 as the WWN of the new LUN.
If the new LUN doesn't appear, confirm that the SAN is properly configured and the HBA port is registered for the new LUN, then re-run both the rescan-scsi-bus.sh --nosync -f -r -m and rescan-scsi-bus.sh -a scripts. If the new LUN doesn't appear, you must reboot the appliance.

Return to the CLI:

[root@solace-primary ~]# exit
[support@solace-primary ~]$ exit

To detect the new LUN on the backup appliance, perform the following steps:

Enter the following command to elevate to the support user, and then enter the support user's password when prompted.
```
solace-backup# shell redundantLunMigration
login: support
Password:
```
Enter the following command to elevate to the root user, and then enter the root user's password when prompted:
```
[support@solace-backup ~]$ su -
Password:
```

Check that the current LUN is visible.

[root@solace-backup ~]# multipath -ll
3600601604d301c008e291bb6a6d6e811 dm-0 DGC     ,RAID 0 size=300G features='2 queue_if_no_path

. . .

[root@solace-backup ~]#

Rescan the SCSI bus to add the new LUN.

[root@solace-backup ~]# rescan-scsi-bus.sh --nosync -f -r -m
[root@solace-backup ~]# rescan-scsi-bus.sh -a

Check that the new LUN has been added.
```
[root@solace-backup ~]# sudo multipath -ll
3600601604d301c008e291bb6a6d6e811 dm-0 DGC     ,RAID 0 size=300G features='2 queue_if_no_path

. . .

360014057d24f4b77681435faf684d587 dm-3 LIO-ORG ,sdc1-4 size=800G features='1 queue_if_no_path' hwhandler='0' wp=rw

. . .

[root@solace-backup ~]#
```
If the new LUN doesn't appear, confirm that the SAN is properly configured and the HBA port is registered for the new LUN, then re-run both the rescan-scsi-bus.sh --nosync -f -r -m and rescan-scsi-bus.sh -a scripts. If the new LUN doesn't appear, you must reboot the appliance.

Return to the CLI:

[root@solace-backup ~]# exit
[support@solace-backup ~]$ exit

Partition and create the file system on the new LUN. For details on how to do this, refer to Configuring an External Disk Array for Guaranteed Messaging.

On both the primary and backup appliances, run the following command to confirm that the new external disk LUN is available and referred to by the correct WWN (obtain the WWN of the new LUN from your storage administrator):

solace-primary> show hardware details

solace-backup> show hardware details

Don't proceed further until the new LUN is visible.

Ensure that message-spool defragmentation is not active.
Enter the following command on both the primary and backup appliances.
```
solace-primary> show message-spool

. . .
            
Defragmentation Status:                   Idle

. . .
```
```
solace-backup> show message-spool

. . .

Defragmentation Status:                   Idle

. . .
```
If the message spool defragmentation status is not Idle for both the primary and backup appliances, wait for the defragmentation process to complete before proceeding.

If Config-Sync is in use, on both the primary and backup appliances enter the following command to confirm that Config-Sync is operationally ready:

solace-primary> show config-sync
Admin Status:                                    Enabled
Oper Status:                                     Up

solace-backup> show config-sync
Admin Status:                                    Enabled
Oper Status:                                     Up

To prevent loss of configuration, do not proceed further if Config-Sync is operationally down.

To stop providing service to applications, enter the following commands on the backup appliance:

solace-backup> enable
solace-backup# configure
solace-backup(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y

Repeat the previous step on the primary appliance:

solace-primary> enable
solace-primary# configure
solace-primary(configure)# service msg-backbone shutdown
All clients will be disconnected.
Do you want to continue (y/n)? y

On the primary appliance, run the show message-spool detail command to confirm that:

the appliance is still active, up, and synchronized (indicated by the Operational, Datapath, and Synchronization statuses)
the appliance has successfully cleaned up all flows (indicated by zeroes in the Currently Used column)

To stop Guaranteed Messaging, enter the following commands on the primary appliance:

solace-primary(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y
solace-primary(configure)# end

Repeat the previous step on the backup appliance:

solace-backup(configure)# hardware message-spool shutdown
All message spooling will be stopped.
Do you want to continue (y/n)? y
solace-backup(configure)# end

On the primary appliance, perform the following steps to migrate the LUN data:

If you do not perform this step, you will have to reset the message spool on the primary appliance after you edit the maximum spool usage later in this procedure. Resetting the message spool causes all guaranteed messaging data to be lost.

Enter the following command to elevate to the support user, and then enter the support user's password when prompted:
```
solace-primary# shell redundantLunMigration
login: support
Password:
```
Enter the following command to elevate to the root user, and then enter the root user's password when prompted:
```
[support@solace-primary ~]$ su -
Password:
```

Migrate the AD keys within p1 and p2 of the old LUN to the new LUN, using the adkey-tool script.

These partitions are located in /dev/mapper/ and are named <wwn><p#>.

[root@solace-primary ~]# adkey-tool migrate --src-device /dev/mapper/<old LUN wwn>p1 --dest-device /dev/mapper/<new LUN wwn>p1
[root@solace-primary ~]# adkey-tool migrate --src-device /dev/mapper/<old LUN wwn>p2 --dest-device /dev/mapper/<new LUN wwn>p2

For example:

[root@solace-primary ~]# adkey-tool migrate --src-device /dev/mapper/3600601604d301c008e291bb6a6d6e811p1 --dest-device /dev/mapper/360014057d24f4b77681435faf684d587p1
[root@solace-primary ~]# adkey-tool migrate --src-device /dev/mapper/3600601604d301c008e291bb6a6d6e811p2 --dest-device /dev/mapper/360014057d24f4b77681435faf684d587p2

The LUN's WWN might be prefixed by a 3 in /dev/mapper.

For example, from earlier in this procedure, the WWN of the new LUN is 60:01:40:57:d2:4f:4b:77:68:14:35:fa:f6:84:d5:87. This WWN appears as /dev/mapper/360014057d24f4b77681435faf684d587p1 and /dev/mapper/360014057d24f4b77681435faf684d587p2 in the example above.

Create the following temporary directories:

[root@solace-primary ~]# mkdir -p /tmp/old_lun_p1
[root@solace-primary ~]# mkdir -p /tmp/old_lun_p2
[root@solace-primary ~]# mkdir -p /tmp/new_lun_p1
[root@solace-primary ~]# mkdir -p /tmp/new_lun_p2

Mount the partition one (p1) and partition two (p2) of both the old and new LUNs.

[root@solace-primary ~]# mount /dev/mapper/<old LUN wwn>p1 /tmp/old_lun_p1
[root@solace-primary ~]# mount /dev/mapper/<old LUN wwn>p2 /tmp/old_lun_p2
[root@solace-primary ~]# mount /dev/mapper/<new LUN wwn>p1 /tmp/new_lun_p1
[root@solace-primary ~]# mount /dev/mapper/<new LUN wwn>p2 /tmp/new_lun_p2

For example:

[root@solace-primary ~]# mount /dev/mapper/3600601604d301c008e291bb6a6d6e811p1 /tmp/old_lun_p1
[root@solace-primary ~]# mount /dev/mapper/3600601604d301c008e291bb6a6d6e811p2 /tmp/old_lun_p2
[root@solace-primary ~]# mount /dev/mapper/360014057d24f4b77681435faf684d587p1 /tmp/new_lun_p1
[root@solace-primary ~]# mount /dev/mapper/360014057d24f4b77681435faf684d587p2 /tmp/new_lun_p2

Copy all directories and files within p1 and p2 of the old LUN to the new LUN:

[root@solace-primary ~]# cp -a /tmp/old_lun_p1/* /tmp/new_lun_p1/
[root@solace-primary ~]# cp -a /tmp/old_lun_p2/* /tmp/new_lun_p2/

Unmount p1 and p2 of both the old and new LUNs:

[root@solace-primary ~]# umount /tmp/old_lun_p1
[root@solace-primary ~]# umount /tmp/old_lun_p2
[root@solace-primary ~]# umount /tmp/new_lun_p1
[root@solace-primary ~]# umount /tmp/new_lun_p2

Return to the CLI:

[root@solace-primary ~]# exit
[support@solace-primary ~]$ exit

Enter the following commands on the primary appliance and then on the backup appliance to configure the message spool to use the new external disk LUN:

solace-primary# configure
solace-primary(configure)# hardware message-spool disk-array wwn <new LUN wwn>

solace-backup# configure
solace-backup(configure)# hardware message-spool disk-array wwn <new LUN wwn>

Where <new LUN wwn> is the WWN of the new LUN, as displayed earlier in this procedure.

To start Guaranteed Messaging and message spooling, enter the following command on the primary appliance, and then on the backup appliance:
```
solace-primary(configure)# no hardware message-spool shutdown primary
```
```
solace-backup(configure)# no hardware message-spool shutdown backup
```
To start providing service to applications, enter the following commands on the primary appliance, and then on the backup appliance:
```
solace-primary(configure)# no service msg-backbone shutdown
```
```
solace-backup(configure)# no service msg-backbone shutdown
```
If the Config-Sync feature is in use, enter the following command on both the primary appliance and backup appliances to verify that Config-Sync is operationally up:
```
solace-primary> show config-sync
Admin Status:                                    Enabled
Oper Status:                                     Up
```
```
solace-backup> show config-sync
Admin Status:                                    Enabled
Oper Status:                                     Up
```
If Config-Sync does not come up, either there were configuration changes performed beyond what is described in this procedure, or one or more steps did not complete as expected. To prevent the configuration from diverging further, immediately investigate and resolve this issue.
Optional: To edit the maximum spool usage for the new LUN, enter the following command, first on the primary appliance and then on the backup appliance:
```
solace-primary(configure)# hardware message-spool max-spool-usage <size>
```
```
solace-backup(configure)# hardware message-spool max-spool-usage <size>
```
Where <size> is the maximum spool usage in megabytes.
- Refer to Configuring Max Spool Usage to set the maximum spool usage.
- If you used the defaults while creating the partitions and file systems earlier in this procedure, the LUN will be split into two equal sized partitions. The max-spool usage on each appliance should be set to 45% of the size of the new LUN.

At this point the LUN migration has succeeded. After the storage administrator has de-provisioned the original LUN, the show hardware details command shows that the old LUN has a state of Down.

Also, the kernel log of the appliances might contain entries reflecting that the old LUN is no longer visible.

Example Log Entries:

2016-03-01T02:43:04+0000 <daemon.notice> solace-primary multipathd: 3600601604d301c008e291bb6a6d6e811: sdd - emc_clariion_checker: Logical Unit is unbound or LUNZ
2016-03-01T02:43:04+0000 <daemon.notice> solace-primary multipathd: 3600601604d301c008e291bb6a6d6e811: sdf - emc_clariion_checker: Logical Unit is unbound or LUNZ

There is no operational impact from these log entries and the Down state of the attached device.

Remove the original LUN from the primary appliance.

Enter the following command to elevate to the support user, and then enter the support user’s password when prompted.
```
solace-primary# shell redundantLunMigration
login: support
Password:
```
Enter the following command to elevate to the root user, and then enter the root user's password when prompted:
```
[support@solace-primary ~]$ su -
Password:
```

Check that both the original and new LUNs are visible.

[root@solace-primary ~]# multipath -ll
3600601604d301c008e291bb6a6d6e811 dm-0 DGC     ,RAID 0 size=300G features='1 retain_attached_hw_handler' hwhandler='1

. . .

360014057d24f4b77681435faf684d587 dm-3 LIO-ORG ,sdc1-4 size=800G features='1 queue_if_no_path' hwhandler='0' wp=rw

. . .

Rescan the SCSI bus to remove the original LUN.
```
[root@solace-primary ~]# rescan-scsi-bus.sh -r
```

Check that the original LUN is removed.

[root@solace-primary ~]# multipath -ll
360014057d24f4b77681435faf684d587 dm-3 LIO-ORG ,sdc1-4 size=800G features='1 queue_if_no_path' hwhandler='0' wp=rw

Return to the CLI:

[root@solace-primary ~]# exit
[support@solace-primary ~]$ exit

Remove the original LUN from the backup appliance.

Enter the following command to elevate to the support user, and then enter the support user's password when prompted.
```
solace-backup# shell redundantLunMigration
login: support
Password:
```
Enter the following command to elevate to the root user, and then enter the root user's password when prompted:
```
[support@solace-backup ~]$ su -
Password:
```

Check that both the original and new LUNs are visible.

[root@solace-backup ~]# multipath -ll
3600601604d301c008e291bb6a6d6e811 dm-0 DGC     ,RAID 0 size=300G features='1 retain_attached_hw_handler' hwhandler='1

. . .

360014057d24f4b77681435faf684d587 dm-3 LIO-ORG ,sdc1-4 size=800G features='1 queue_if_no_path' hwhandler='0' wp=rw

. . .

Rescan the SCSI bus to remove the original LUN.
```
[root@solace-backup ~]# rescan-scsi-bus.sh -r
```

Check that the original LUN is removed.

[root@solace-backup ~]# multipath -ll
360014057d24f4b77681435faf684d587 dm-3 LIO-ORG ,sdc1-4 size=800G features='1 queue_if_no_path' hwhandler='0' wp=rw

Return to the CLI:

[root@solace-backup ~]# exit
[support@solace-backup ~]$ exit

Provide feedback