docs,how-to: restoring LINSTOR database backups

We automatically create backups, we should also inform people on how to use
them.

Signed-off-by: Moritz Wanzenböck <moritz.wanzenboeck@linbit.com>

docs/how-to/README.md

@@ -29,3 +29,7 @@ These guides show you how to configure a specific aspect or achieve a specific t
 
 * [How to Use the Host Network for DRBD Replication](./drbd-host-networking.md)
 * [How to Use Transport Layer Security for DRBD Replication](./drbd-tls.md)
+
+## Maintenance
+
+* [How to Restore a LINSTOR Database Backup](./restore-linstor-db.md)

docs/how-to/restore-linstor-db.md

@@ -0,0 +1,87 @@
+# How to Restore a LINSTOR Database Backup
+
+This guide shows you how to restore a LINSTOR® Controller from a database backup.
+
+To complete this guide, you should be familiar with:
+
+* using the `kubectl` command line tool to access the Kubernetes cluster.
+
+## Find the Latest Backup
+
+On every Database migration, the LINSTOR Controller deployment will first create a backup. The backup is stored
+in Kubernetes Secrets. List all backups by using the following command:
+
+```
+$ kubectl get secrets --field-selector type=piraeus.io/linstor-backup --sort-by .metadata.creationTimestamp -ocustom-columns="NAME:.metadata.name,CREATED-AT:metadata.creationTimestamp,VERSION:.metadata.annotations.piraeus\.io/linstor-version"
+NAME                                                     CREATED-AT             VERSION
+linstor-backup-for-linstor-controller-db7fbfd95-zsfmm    2024-11-04T07:54:29Z   LINSTOR Controller 1.27.0
+linstor-backup-for-linstor-controller-745d54bf99-544hf   2024-11-04T08:03:59Z   LINSTOR Controller 1.29.1
+```
+
+Select the backup you want to restore, making note of the name. For example, to restore the LINSTOR 1.27.0 version,
+we set:
+
+```
+$ BACKUP_NAME=linstor-backup-for-linstor-controller-db7fbfd95-zsfmm
+```
+
+## Temporarily stop the LINSTOR Controller Deployment
+
+To perform the upgrade, ensure that the LINSTOR Controller is shut down by scaling the Piraeus Operator and
+LINSTOR Controller deployment to 0 replicas:
+
+```
+$ kubectl scale deployment piraeus-operator-controller-manager --replicas 0
+deployment.apps/piraeus-operator-controller-manager scaled
+$ kubectl scale deployment linstor-controller --replicas 0
+deployment.apps/linstor-controller scaled
+```
+
+## Create a New Backup of the Current Cluster State
+
+Since the restore process is destructive, first create a backup of the current database:
+
+```
+$ mkdir backup
+$ cd backup
+$ kubectl api-resources --api-group=internal.linstor.linbit.com -oname | xargs --no-run-if-empty kubectl get crds -oyaml > crds.yaml
+$ kubectl api-resources --api-group=internal.linstor.linbit.com -oname | xargs --no-run-if-empty -I {} sh -c 'kubectl get {} -oyaml > {}.yaml'
+```
+
+## Restore the Database
+
+Copy and unpack the selected backup to a local directory by using the following commands:
+
+```
+$ mkdir restore
+$ cd restore
+# Replace this with your selected backup name
+$ kubectl get secrets --sort-by=.metadata.name -l piraeus.io/backup=$BACKUP_NAME -ogo-template='{{range .items}}{{index .data "backup.tar.gz" | base64decode}}{{end}}' > backup.tar.gz
+$ tar -xvf backup.tar.gz
+crds.yaml
+[...]
+```
+
+Then, replace the current database with the database from the backup:
+
+```
+$ kubectl api-resources --api-group=internal.linstor.linbit.com -oname | xargs --no-run-if-empty kubectl delete crds
+$ kubectl create -f .
+```
+
+## Restart the LINSTOR Controller Deployment
+
+Now we can safely restart the Piraeus Operator and LINSTOR Controller Deployment.
+
+```
+$ kubectl scale deployment piraeus-operator-controller-manager --replicas 1
+deployment.apps/piraeus-operator-controller-manager scaled
+$ kubectl scale deployment linstor-controller --replicas 1
+deployment.apps/linstor-controller scaled
+$ kubectl rollout status deployment piraeus-operator-controller-manager
+Waiting for deployment "piraeus-operator-controller-manager" rollout to finish: 0 of 1 updated replicas are available...
+deployment "piraeus-operator-controller-manager" successfully rolled out
+$ kubectl rollout status deployment linstor-controller
+Waiting for deployment "linstor-controller" rollout to finish: 0 of 1 updated replicas are available...
+deployment "linstor-controller" successfully rolled out
+```