Merge pull request #537 from PAWECOGmbH/staging

Staging to production

compose-dev.yml

@@ -1,6 +1,6 @@
 services:
   lucee:
-    image: ${LUCEE_IMAGE}
+    image: ${LUCEE_IMAGE}:${LUCEE_IMAGE_VERSION}
     ports:
       - "${LUCEE_PORT}:80"
     restart: always

compose.yml

@@ -1,6 +1,6 @@
 services:
   lucee:
-    image: ${LUCEE_IMAGE}
+    image: ${LUCEE_IMAGE}:${LUCEE_IMAGE_VERSION}
     ports:
       - "${LUCEE_PORT}:80"
     restart: always

config/backup/backup.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Enable automatic export of variables
+set -a
+
+# Load the .env file from the project root
+source "$(dirname "$0")/../../.env"
+
+# Dynamically generate volume names based on the project
+DB_VOLUME="${COMPOSE_PROJECT_NAME}_db_volume"
+USERDATA_VOLUME="${COMPOSE_PROJECT_NAME}_userdata_volume"
+
+# Date format for versioning (e.g., 20241022_2300)
+TIMESTAMP=$(date +"%Y%m%d_%H%M")
+
+# Check if the /backup folder exists and create it if necessary
+if [ ! -d "/backup" ]; then
+    mkdir -p /backup
+fi
+
+# Create remote directories if they don't exist
+ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "mkdir -p ${REMOTE_BACKUP_PATH}/db ${REMOTE_BACKUP_PATH}/userdata ${REMOTE_BACKUP_PATH}/lucee"
+
+# Backup database volume and store in the remote db directory
+docker run --rm -v ${DB_VOLUME}:/volume -v /backup:/backup alpine sh -c "tar -czf /backup/database_${TIMESTAMP}.tar.gz -C /volume ."
+scp -i ${SSH_KEY_PATH} /backup/database_${TIMESTAMP}.tar.gz ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/db/
+
+# Backup userdata volume and store in the remote userdata directory
+docker run --rm -v ${USERDATA_VOLUME}:/volume -v /backup:/backup alpine sh -c "tar -czf /backup/userdata_${TIMESTAMP}.tar.gz -C /volume ."
+scp -i ${SSH_KEY_PATH} /backup/userdata_${TIMESTAMP}.tar.gz ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/userdata/
+
+# Backup Lucee image and store in the remote lucee directory
+docker save -o /backup/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}_${TIMESTAMP}.tar ${LUCEE_IMAGE}:${LUCEE_IMAGE_VERSION}
+scp -i ${SSH_KEY_PATH} /backup/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}_${TIMESTAMP}.tar ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/lucee/
+
+# Rotate backups: Keep only the latest 30 backups per type
+
+# For database backups
+ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "cd ${REMOTE_BACKUP_PATH}/db && ls -tp | grep -v '/$' | tail -n +31 | xargs -I {} rm -- {}"
+
+# For userdata backups
+ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "cd ${REMOTE_BACKUP_PATH}/userdata && ls -tp | grep -v '/$' | tail -n +31 | xargs -I {} rm -- {}"
+
+# For Lucee image backups
+ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "cd ${REMOTE_BACKUP_PATH}/lucee && ls -tp | grep -v '/$' | tail -n +31 | xargs -I {} rm -- {}"
+
+# Disable automatic export of variables
+set +a

config/backup/readme.md

@@ -0,0 +1,74 @@
+
+# Backup for the Production and Staging Environments
+
+**Purpose**
+
+This directory contains the necessary configurations and scripts for backing up and restoring the **database**, **user data**, and the **Lucee image** in the **production** and **staging** environments. The backup and restore processes are automated using Docker and shell scripts to ensure consistency, reliability, and minimal manual intervention.
+
+Note: These scripts are not intended for use in the **development** environment.
+
+
+## **Structure**
+
+-   **backup.sh**: A shell script that automates the process of creating backups for the **database volume**, **user data volume**, and **Lucee image**. Each backup is timestamped to ensure that multiple versions can be maintained. The script also uses `scp` to transfer the backups securely to a remote server.
+
+-   **restore.sh**: A shell script that automates the process of restoring backups from the remote server. It retrieves the backups for the **database**, **user data**, and **Lucee image**, and restores them to the appropriate Docker volumes.
+
+-   **.env**: Contains environment variables required for the backup and restore processes, such as volume names, SSH key, server IP, and remote paths. The backup does not have its own `.env` file; instead, it uses the `.env` file from the main project.
+
+## **Usage**
+
+### **Backup**
+
+To create a backup of the **database**, **user data**, and **Lucee image**, follow these steps:
+
+1.  Ensure that the `.env` file is correctly configured with your **production** or **staging** environment settings (e.g., volume names, remote server path, SSH keys, and server IP).
+
+2.  Navigate to the `config/backup/` directory:
+    `cd config/backup/`
+
+3. Run the backup script:
+`bash backup.sh`
+
+This will:
+
+-   Backup the **database volume**.
+-   Backup the **user data volume**.
+-   Backup the **Lucee image**.
+-   Securely transfer all backups to the remote backup server.
+
+Each backup will be **timestamped** in the format `YYYYMMDD_HHMM`, ensuring you can differentiate between multiple backup versions.
+
+### **Restore**
+
+To restore from a backup, first navigate to the `config/backup/` directory:
+`cd config/backup/`
+
+Then, if you run the restore script without any parameters:
+`bash restore.sh`
+
+It will display a list of available options, such as:
+Usage: restore.sh [--db [TIMESTAMP]] [--userdata [TIMESTAMP]] [--lucee-image [TIMESTAMP]] [--list]
+
+To perform a restore, you need to specify which backup you want to restore by using one of the following options:
+ - To restore the **latest** backup for the **database**:
+ `bash restore.sh --db`
+
+ - To restore a **specific** database backup by **timestamp**:
+ `bash restore.sh --db 20241019_2300`
+
+- To list all available backups on the remote server:
+ `bash restore.sh --list`
+
+
+## **Automating Backups**
+
+To automate the backup process, you can set up a **cron job** to run the backup script at regular intervals (e.g., daily). For example, to run the backup every night at midnight, add the following entry to your crontab:
+`0 0 * * * /path/to/your/project/config/backup/backup.sh`
+
+
+## **Notes**
+
+-   These backups are intended for the **production** and **staging** environments. Ensure that the environment variables in the `.env` file are correctly configured before running any backups or restores.
+-   The backup script automatically **rotates** backups, keeping only the **latest 30 backups** per backup type (database, user data, Lucee image) by removing older backups on the remote server.
+-   Always ensure that your **SSH keys** and **server information** are secure, as they are used for transferring backups between the production or staging environment and the remote server.

config/backup/restore.sh

@@ -0,0 +1,93 @@
+#!/bin/bash
+
+# Enable automatic export of variables
+set -a
+
+# Load the .env file from the project root
+source "$(dirname "$0")/../../.env"
+
+# Dynamically generate volume names based on the project
+DB_VOLUME="${COMPOSE_PROJECT_NAME}_db_volume"
+USERDATA_VOLUME="${COMPOSE_PROJECT_NAME}_userdata_volume"
+
+# Check if the /restore folder exists and create it if necessary
+if [ ! -d "/restore" ]; then
+    mkdir -p /restore
+fi
+
+# Functions for the various restore processes
+
+restore_db() {
+    if [ -z "$1" ]; then
+        echo "Restoring the latest database backup..."
+        # Get the latest database backup
+        LATEST_DB_BACKUP=$(ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls -t ${REMOTE_BACKUP_PATH}/db | head -n 1")
+    else
+        echo "Restoring database backup from $1..."
+        LATEST_DB_BACKUP="database_$1.tar.gz"
+    fi
+    scp -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/db/${LATEST_DB_BACKUP} /restore/database.tar.gz
+    docker run --rm -v ${DB_VOLUME}:/volume -v /restore:/restore alpine sh -c "tar -xzf /restore/database.tar.gz -C /volume"
+    docker restart ${MYSQL_CONTAINER_NAME}
+}
+
+restore_userdata() {
+    if [ -z "$1" ]; then
+        echo "Restoring the latest userdata backup..."
+        # Get the latest userdata backup
+        LATEST_USERDATA_BACKUP=$(ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls -t ${REMOTE_BACKUP_PATH}/userdata | head -n 1")
+    else
+        echo "Restoring userdata backup from $1..."
+        LATEST_USERDATA_BACKUP="userdata_$1.tar.gz"
+    fi
+    scp -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/userdata/${LATEST_USERDATA_BACKUP} /restore/userdata.tar.gz
+    docker run --rm -v ${USERDATA_VOLUME}:/volume -v /restore:/restore alpine sh -c "tar -xzf /restore/userdata.tar.gz -C /volume"
+    docker restart ${LUCEE_CONTAINER_NAME}
+}
+
+restore_lucee_image() {
+    if [ -z "$1" ]; then
+        echo "Restoring the latest Lucee image backup..."
+        # Get the latest Lucee image backup
+        LATEST_LUCEE_BACKUP=$(ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls -t ${REMOTE_BACKUP_PATH}/lucee | head -n 1")
+    else
+        echo "Restoring Lucee image backup from $1..."
+        LATEST_LUCEE_BACKUP="image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}_$1.tar"
+    fi
+    scp -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP}:${REMOTE_BACKUP_PATH}/lucee/${LATEST_LUCEE_BACKUP} /restore/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}.tar
+    docker load -i /restore/image_${LUCEE_IMAGE}_${LUCEE_IMAGE_VERSION}.tar
+}
+
+list_backups() {
+    echo "Available backups on remote server:"
+    echo "Database backups:"
+    ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls ${REMOTE_BACKUP_PATH}/db"
+    echo ""
+    echo "Userdata backups:"
+    ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls ${REMOTE_BACKUP_PATH}/userdata"
+    echo ""
+    echo "Lucee image backups:"
+    ssh -i ${SSH_KEY_PATH} ${SERVER_USER}@${SERVER_IP} "ls ${REMOTE_BACKUP_PATH}/lucee"
+    echo ""
+}
+
+# Show help if no options have been specified
+if [ $# -eq 0 ]; then
+    echo "Usage: $0 [--db [TIMESTAMP]] [--userdata [TIMESTAMP]] [--lucee-image [TIMESTAMP]] [--list]"
+    exit 1
+fi
+
+# Process the specified options
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --db) restore_db "$2"; shift ;;
+        --userdata) restore_userdata "$2"; shift ;;
+        --lucee-image) restore_lucee_image "$2"; shift ;;
+        --list) list_backups; exit 0 ;;
+        *) echo "Unknown option: $1"; exit 1 ;;
+    esac
+    shift
+done
+
+# Disable automatic export of variables
+set +a

config/example.env

@@ -1,8 +1,11 @@
 ## Project name
 COMPOSE_PROJECT_NAME=saaster # Must be unique on docker host.
 
-## Lucee settings (default: lucee/lucee:6.0-nginx)
-LUCEE_IMAGE=lucee/lucee:6.0-nginx # For the initial setup, please leave it as is, then use your own image name afterward.
+## Lucee settings
+# Keep the Lucee image name for setup, then replace with your own later.
+# Default: lucee/lucee:6.0-nginx
+LUCEE_IMAGE=lucee/lucee
+LUCEE_IMAGE_VERSION=6.0-nginx
 LUCEE_CONTAINER_NAME=saaster_lucee # Must be unique on docker host.
 LUCEE_PORT=8080 # Must be unique on docker host.
 LUCEE_ADMIN_PASSWORD=defaultpass
@@ -22,8 +25,19 @@ FLYWAY_DB_FOLDER=core # core or myapp
 FLYWAY_MIGRATION_TYPE=migrate # migrate or repair
 FLYWAY_CONTAINER_NAME=saaster_flyway # Must be unique on docker host.
 
-## Inbucket settings (dev)
+## Inbucket settings (development environment)
 INBUCKET_CONTAINER_NAME=saaster_inbucket # Will be used as the SMTP server in Lucee.
 INBUCKET_SMTP_PORT=2500 # Must be unique on docker host. In Lucee Admin please set to 2500.
 INBUCKET_WEB_PORT=9000 # Must be unique on docker host.
-INBUCKET_POP3_PORT=1100 # Must be unique on docker host.
+INBUCKET_POP3_PORT=1100 # Must be unique on docker host.
+
+
+## Backup settings
+
+# Backup path on the remote server
+REMOTE_BACKUP_PATH=/backups
+
+# Server information for SCP Transfer
+SERVER_USER=root
+SERVER_IP=xxx.xxx.xxx.xxx
+SSH_KEY_PATH=~/.ssh/id_rsa