Skip to content

Disaster Recovery

Jump to bottom
Daniel Glen edited this page Apr 2, 2024 · 18 revisions

Runbooks

I want to restore a file to a previous version

You need to rollback to the version you would like to go back to.

  1. Verify on the AWS console that a backup version exists
  2. Connect to the service pod $ kubectl exec -n $namespace --stdin --tty $pod_name -- /bin/sh
  3. Once connected, run aws s3api list-object-versions --bucket bucketname --prefix path/to/file. This will list all versions of the file in chronological order by LastModified date.
  4. The next step is to delete all previous versions until you get to the one you would like to roll back to.
  5. To delete a version run aws s3api delete-object --bucket bucketname --key path/to/file --version-id versionId

I want to restore my database to a previous snapshot

Snapshots are currently taken daily and can also be manually taken whenever necessary

  1. Get the db-instance-identifier of the DB instance you want to restore, using the Cloud Platform CLI; cloud-platform decode-secret -n hale-platform-env -s rds-instance-output

  2. Look for a line containing ‘rds_instance_address’ E.g cloud-platform-gffgfg.hthf.eu-west-2.rds.amazonaws.com

  3. Search for snapshots of the db using ...

  4. Locate the DBSnapshotIdentifier of the snapshot you would like to restore to. E.g rds:cloud-platform-gffgfg.hthf-2024-03-20-06-18

  5. Update the rds.tf in cloud-platform-environments and the hale environment you would like to restore to.

  6. Add the line snapshot_identifier = "rds:cloud-platform-gffgfg.hthf-2024-03-20-06-18"

  7. Your RDS db_allocated_storage will be autoscaled to maximum of 10000 by default unless you have set to specific number. Hence, the actual db_allocated_storage may be higher than what is set as default(10).

  8. Check the actual “AllocatedStorage” value described in your RDS instance using the command aws rds describe-db-instances --db-instance-identifier <your-rds-db-identifier> This was located in step 4.

  9. If that is different to what is set for your RDS, update your RDS manifest to include the actual db_allocated_storage by adding db_allocated_storage = "<the actual value you got from the above cli command>"

Clone this wiki locally