8.4 KiB
subtitle | header-img | date | description | related | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Upgrades | img/header_img/server.jpg | 2018-03-07 09:50:12 | Upgrading Spectrum Protect Server in a Docker Container |
|
Upgrading Spectrum Protect Server (in a container)
NOTES:
- The storage used in the example below is from the host file systems, passed to the container with Docker's
-v
option.- Upgrading needs special thought. I have found that upgrading within a major release, EG: 7.1.0 to 7.1.1, 7.1.2 to 7.1.3, etc works transparently. If upgrading that results in a version of DB2 changing (like v7 to v8) additional consideration needs to be made that has not yet been captured in my
upgrade
option below. (Upgrading from v7 to v8 failed for me because of DB2 issues that I wasn't able to resolve - DB2 changed versions between 7 and 8.) After the upgrade Spectrum Protect is smart enough to realise that it has been upgraded and will do a one time schema update.- As always, make sure you have a back-out plan, in case things go pair shaped. Docker itself provides a great back-out option, especially if you use it with persistent Docker Container storage that can exploit snapshots and reverting (or directly accessing) a snapshot. In the example below, I'm using Linux LVM snapshots to provide my backup option. Spectrum Scale would also be a good choice.
Configuration
Our configuration will use these defaults
TSM_DB=/srv/sp/database
TSM_USER=/srv/sp/user
TSM_DATA=/srv/sp/data
- TSM_DB is where the TSM Database will be stored on the Docker Host, and will be passed to the container with
-v $TSM_DB:/database
- TSM_USER is where the TSM Instance user's home directory is on the Docker Host, and will be passed to the container with
-v $TSM_USER:/tsm
- TSM_DATA is where the TSM Disk based storage pools are stored on the Docker Host, and will be passed to the container with
-v $TSM_DATA:/data
If data is stored in a cloud pool, then this would be where the acceleration pool is holding data before it gets sent to the cloud.
Upgrade Steps
Backup Spectrum Protect's Database
backup db type=FULL devclass=<YOUR BACKUP DEVCLASS> protectk=y pass=<PASSWORD> wait=yes
Protect: SERVER1>backup db t=f devclass=dbb protectk=y pass=password wait=y
ANR0984I Process 143 for Database Backup started in the FOREGROUND at 14:48:02.
ANR4559I Backup DB is in progress.
ANR2280I Full database backup started as process 143.
ANR4626I Database backup will use 1 streams for processing with the number originally requested 1.
ANR4550I Full database backup (process 143) completed. Total bytes backed up: 2,215,641,088.
ANR0985I Process 143 for Database Backup running in the FOREGROUND completed with completion state SUCCESS at 14:52:45.
ANR2394I BACKUP DB: Server device configuration information was written to all device configuration files.
Protect: SERVER1>q volh type=dbb
...
Date/Time: 14-03-2018 14:48:04
Volume Type: BACKUPFULL
Backup Series: 253
Backup Operation: 0
Volume Seq: 100,001
Device Class: DBB
Volume Name: /data/dbb/20999284.DBV
Make sure your backup is successful and make a note of where it is, in case you need it.
[root@host]# ls -aln $TSM_DATA/dbb/|grep 20999284
-rw------- 1 201 201 2215642514 Mar 14 14:51 20999284.dbv
Stop Spectrum Protect
Stop your SP server.
Protect: SERVER1>halt
ANR2234W This command will halt the server; if the command is issued from a remote client, it may not be possible to restart the server from the remote location.
Do you wish to proceed? (Yes (Y)/No (N)) y
ANS8002I Highest return code was 0.
[root@host]#
NOTE: Don't use docker stop
to stop your SP server.
By default docker sends a SIGTERM signal to the containers process 1 and then waits 10 seconds for it to die. If it hasn't died, it will then send it a SIGKILL.
While you can control the timeout between the SIGTERM and SIGKILL signal and you can even control which kill signal is sent, I haven't explored what signals SP will act on. So as a recommendation, always use SP halt
command to stop an SP server.
If you can, snapshot your persistent storage.
If you can snapshot your persistent storage that you provide to the SP Docker Container, that will save you doing a "restoredb" if something goes pair shaped.
[root@host]# df /srv/sp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vgroot-lvprotect 20511356 10031500 9431280 52% /srv/sp
[root@host]# lvcreate -s -L5G -n lvspsnapshot vgroot/lvprotect
Using default stripesize 64.00 KiB.
Logical volume "lvspsnapshot" created.
NOTE: If you are using IBM Spectrum Scale, you can use mmcrsnapshot
Delete your container
docker rm -v spectrumprotect
Start the container with the new version of SP
You need to do a one-time start of the container with the option upgrade
.
docker run --detach=true --hostname=tsm --interactive=false -v $TSM_USER:/tsm -v $TSM_DB:/database --privileged=true --rm=false --tty=false --name=spectrumprotect ibm/spectrumprotect:8.1.4 upgrade
NOTE: You can see the console by running docker logs -f spectrumprotect
to make sure the upgrade works as expected. Once you see success, you can delete the container with docker rm -v spectrumprotect
.
[root@host]# docker logs -f spectrumprotect
DB2 installation is being initialized.
Total number of tasks to be performed: 4
Total estimated time for all tasks to be performed: 309 second(s)
Task #1 start
Description: Setting default global profile registry variables
Estimated time 1 second(s)
Task #1 end
...
The execution completed successfully.
For more information see the DB2 installation log at "/tmp/db2iupdt.log.5".
DBI1446I The db2iupdt command is running.
DBI1070I Program db2iupdt completed successfully.
Normal Run
Spectrum Protect is ready to run on the new version as normal. The first time it runs at a later release, you should see some ANR0276I
messages.
docker run --detach=true --hostname=tsm --interactive=false -v $TSM_USER:/tsm -v $TSM_DB:/database -v $TSM_DATA:/data --memory=16g -p 1500:1500 -p 1543:1543 --privileged=true --rm=false --restart=no --tty=false --name=spectrumprotect ibm/spectrumprotect:8.1.4
ANR7800I DSMSERV generated at 16:03:41 on Nov 17 2017.
IBM Spectrum Protect for Linux/x86_64
Version 8, Release 1, Level 4.000
Licensed Materials - Property of IBML
(C) Copyright IBM Corporation 1990, 2017.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.
ANR7801I Subsystem process ID is 7.
...
ANR0275I Detected schema change for table SD.Pool.Clusters, attempting to
reconcile.
ANR0276I Schema reconciliation for table SD.Pool.Clusters succeeded.
ANR0275I Detected schema change for table SD.Containers, attempting to
reconcile.
ANR0276I Schema reconciliation for table SD.Containers succeeded.
ANR0275I Detected schema change for table SD.Replicated.Stgpools, attempting to
reconcile.
ANR0276I Schema reconciliation for table SD.Replicated.Stgpools succeeded.
...
IBM Spectrum Protect:SERVER1>
I did this on the lab server, and while admittedly it doesn't have much data - the whole "upgrade" process took 12 minutes after the DB backup was complete - and that was with me documenting it for this wiki. So it could have been quicker!
Happy days :)
If something did go pear shaped, then revert your snapshot (or if it is read only, copy the files back to the original path) and then start your container with the original image (the old version) of SP. It should start up as normal as if it was only shut down temporarily. If you really have problems, you have that backup to recover from! Start the container with the
restore <PASSWORD>
option - you'll need a DBDIRS.TXT file as well in the instance home directory. More info on recovering here.