docker-volume-backup.archive-pre get executed in each container

[alexander-zimmermann]

Hey,

I setup my backup like discussed in #123. Works as expected. To further improve my backup strategy, I would like to dump my DBs via docker-volume-backup.archive-pre. For example I add the 2 labels to my marinaDB container.

labels:
  - docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump -p $MYSQL_ROOT_PASSWORD --all-databases > /config/backup.sql'
  - docker-volume-backup.archive-post=/bin/sh -c 'rm /config/backup.sql'

My assumption was that the backup only executed this in the marinaDB container (since I add the label only there :-)). However, the logs say something different...

crond: USER root pid 199 cmd /bin/sh -c 'set -a; source /etc/dockervolumebackup/conf.d/gollum_daily.env; set +a && backup' 2>&1
time="2022-07-14T02:00:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'influx backup --compression gzip /var/lib/influxdb2/backup' for container influx"
time="2022-07-14T02:00:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'mysqldump -pXXX --all-databases > /config/backup.sql' for container mariadb"
time="2022-07-14T02:00:00Z" level=error msg="Fatal error running backup: withLabeledCommands: archive: error running pre commands: runLabeledCommands: error from errgroup: runLabeledCommands: error executing command: exec: running command exited 1"
crond: USER root pid 206 cmd /bin/sh -c 'set -a; source /etc/dockervolumebackup/conf.d/grafana_daily.env; set +a && backup' 2>&1
time="2022-07-14T02:10:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'mysqldump -pXXX --all-databases > /config/backup.sql' for container mariadb"
time="2022-07-14T02:10:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'influx backup --compression gzip /var/lib/influxdb2/backup' for container influx"
time="2022-07-14T02:10:00Z" level=error msg="Fatal error running backup: withLabeledCommands: archive: error running pre commands: runLabeledCommands: error from errgroup: runLabeledCommands: error executing command: exec: running command exited 1"
crond: USER root pid 213 cmd run-parts /etc/periodic/15min
crond: USER root pid 214 cmd /bin/sh -c 'set -a; source /etc/dockervolumebackup/conf.d/heimdall_daily.env; set +a && backup' 2>&1
time="2022-07-14T02:20:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'mysqldump -p Lrttxr-8QH4goppWtm!U --all-databases > /config/backup.sql' for container mariadb"

[m90]

My assumption was that the backup only executed this in the marinaDB container (since I add the label only there :-)).

So from what I can understand from your log output, you have a mariadb container executing the mysqldump and an influx container that is running influx backup, which seems to be expected behavior. it seems that the mariadb container is trying to run the commands you have given in the labels, nothing is mapped incorrectly.

I'm not sure I understand your question yet, but do you want to prevent the influx command to run on this schedule? In case yes, you can use the EXEC_LABEL option to scope down the number of applicable containers when looking for commands to run.

From the README:

# Without any further configuration, all commands defined in labels will be
# run before and after a backup. If you need more fine grained control, you
# can use this option to set a label that will be used for narrowing down
# the set of eligible containers. When set, an eligible container will also need
# to be labeled as `docker-volume-backup.exec-label=database`.

# EXEC_LABEL="database"

Also see the second section here: https://github.com/offen/docker-volume-backup#run-custom-commands-during-the-backup-lifecycle

[alexander-zimmermann]

No, IMO the mapping is broken. Take a look on the schedule

alexander@ubuntu:~$ docker logs docker-volume-backup 
/etc/dockervolumebackup/conf.d was found, using configuration files from this directory.
Appending cron.d entry with expression 0 2 * * * and configuration file /etc/dockervolumebackup/conf.d/gollum_daily.env
Appending cron.d entry with expression 0 6 1 * * and configuration file /etc/dockervolumebackup/conf.d/gollum_monthly.env
Appending cron.d entry with expression 0 4 * * * and configuration file /etc/dockervolumebackup/conf.d/gollum_weekely.env
Appending cron.d entry with expression 10 2 * * * and configuration file /etc/dockervolumebackup/conf.d/grafana_daily.env
Appending cron.d entry with expression 10 6 1 * * and configuration file /etc/dockervolumebackup/conf.d/grafana_monthly.env
Appending cron.d entry with expression 10 4 * * * and configuration file /etc/dockervolumebackup/conf.d/grafana_weekely.env
Appending cron.d entry with expression 20 2 * * * and configuration file /etc/dockervolumebackup/conf.d/heimdall_daily.env
Appending cron.d entry with expression 20 6 1 * * and configuration file /etc/dockervolumebackup/conf.d/heimdall_monthly.env
Appending cron.d entry with expression 20 4 * * * and configuration file /etc/dockervolumebackup/conf.d/heimdall_weekely.env
Appending cron.d entry with expression 30 2 * * * and configuration file /etc/dockervolumebackup/conf.d/influx_daily.env
Appending cron.d entry with expression 30 6 1 * * and configuration file /etc/dockervolumebackup/conf.d/influx_monthly.env
Appending cron.d entry with expression 30 4 * * * and configuration file /etc/dockervolumebackup/conf.d/influx_weekely.env
Appending cron.d entry with expression 40 2 * * * and configuration file /etc/dockervolumebackup/conf.d/loki_daily.env
Appending cron.d entry with expression 40 6 1 * * and configuration file /etc/dockervolumebackup/conf.d/loki_monthly.env
Appending cron.d entry with expression 40 4 * * * and configuration file /etc/dockervolumebackup/conf.d/loki_weekely.env
Appending cron.d entry with expression 50 2 * * * and configuration file /etc/dockervolumebackup/conf.d/mariadb_daily.env
Appending cron.d entry with expression 50 6 1 * * and configuration file /etc/dockervolumebackup/conf.d/mariadb_monthly.env
Appending cron.d entry with expression 50 4 * * * and configuration file /etc/dockervolumebackup/conf.d/mariadb_weekely.env
Appending cron.d entry with expression 0 3 * * * and configuration file /etc/dockervolumebackup/conf.d/mosquitto_daily.env
Appending cron.d entry with expression 0 7 1 * * and configuration file /etc/dockervolumebackup/conf.d/mosquitto_monthly.env
Appending cron.d entry with expression 0 5 * * * and configuration file /etc/dockervolumebackup/conf.d/mosquitto_weekely.env
Appending cron.d entry with expression 10 3 * * * and configuration file /etc/dockervolumebackup/conf.d/node-red_daily.env
Appending cron.d entry with expression 10 7 1 * * and configuration file /etc/dockervolumebackup/conf.d/node-red_monthly.env
Appending cron.d entry with expression 10 5 * * * and configuration file /etc/dockervolumebackup/conf.d/node-red_weekely.env
Appending cron.d entry with expression 20 3 * * * and configuration file /etc/dockervolumebackup/conf.d/portainer_daily.env
Appending cron.d entry with expression 20 7 1 * * and configuration file /etc/dockervolumebackup/conf.d/portainer_monthly.env
Appending cron.d entry with expression 20 5 * * * and configuration file /etc/dockervolumebackup/conf.d/portainer_weekely.env
Appending cron.d entry with expression 30 3 * * * and configuration file /etc/dockervolumebackup/conf.d/redis_daily.env
Appending cron.d entry with expression 30 7 1 * * and configuration file /etc/dockervolumebackup/conf.d/redis_monthly.env
Appending cron.d entry with expression 30 5 * * * and configuration file /etc/dockervolumebackup/conf.d/redis_weekely.env
Appending cron.d entry with expression 40 3 * * * and configuration file /etc/dockervolumebackup/conf.d/syslog-ng_daily.env
Appending cron.d entry with expression 40 7 1 * * and configuration file /etc/dockervolumebackup/conf.d/syslog-ng_monthly.env
Appending cron.d entry with expression 40 5 * * * and configuration file /etc/dockervolumebackup/conf.d/syslog-ng_weekely.env

At 2am, the first backup is scheduled: gollum. From the first log, you see that crond (pid 199) get executed for gollum. So far so good. But then, it tries to dump the DB...

[m90]

At 2am, the first backup is scheduled: gollum. From the first log, you see that crond (pid 199) get executed for gollum. So far so good. But then, it tries to dump the DB...

This is the expected behavior, unless you use EXEC_LABEL (which I assume you don't). Otherwise, how would the command know which commands apply to which schedule? Hence, it tries to run everything.

It's important to understand that the command has no way of knowing how the volumes/mounts being backed up map to containers. If you backup the volume containing your data, the command does not know which other containers are using the volume.

[alexander-zimmermann]

From the label (like for example traefik do that)?

With EXEC_LABEL how I can achieve that marinadb uses his command and influx another one. Could I add the label to /etc/dockervolumebackup/conf.d files for correct mapping?

[m90]

With EXEC_LABEL how I can achieve that marinadb uses his command and influx another one.

This is explained in the README: https://github.com/offen/docker-volume-backup#run-custom-commands-during-the-backup-lifecycle

In you particular case, you'd have labels like:

labels:
  - docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump -p $MYSQL_ROOT_PASSWORD --all-databases > /config/backup.sql'
  - docker-volume-backup.archive-post=/bin/sh -c 'rm /config/backup.sql'
  - docker-volume-backup.exec-label=mariadb

and then set EXEC_LABEL=mariadb in your mariadb_*.env files.

[alexander-zimmermann]

Thanks for the support. Will test it

[alexander-zimmermann]

Hmm, proposed solution doesn't work. Still it executes the DB dump commands for each volume / container.

Example log from syslog-ng volume/container:

crond: USER root pid 283 cmd /bin/sh -c 'set -a; source /etc/dockervolumebackup/conf.d/syslog-ng_weekely.env; set +a && backup' 2>&1
time="2022-07-15T05:40:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'influx backup --compression gzip /var/lib/influxdb2/backup' for container influx"
time="2022-07-15T05:40:00Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'mysqldump -p XXX --all-databases > /config/backup.sql' for container mariadb"
time="2022-07-15T05:40:00Z" level=error msg="Fatal error running backup: withLabeledCommands: archive: error running pre commands: runLabeledCommands: error from errgroup: runLabeledCommands: error executing command: exec: running command exited 1"

Relevant config:

  # --------------------------------------------------------------------------
  # Mariadb
  # --------------------------------------------------------------------------
  mariadb:
    image: linuxserver/mariadb
    container_name: mariadb
    restart: unless-stopped
    ports:
      - 3306:3306
    volumes:
      - mariadb_data:/config
    environment:
      <<: *default-environment
      PUID: 1000
      PGID: 1000
      MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
    labels:
      - com.centurylinklabs.watchtower.enable=true
      - docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump -p $MYSQL_ROOT_PASSWORD --all-databases > /config/backup.sql'
      - docker-volume-backup.archive-post=/bin/sh -c 'rm /config/backup.sql'
      - docker-volume-backup.exec-label=mariadb
    logging:
      <<: *default-logging

alexander@ubuntu:~/home-automation$ cat docker-volume-backup/config/mariadb_*
BACKUP_SOURCES=/backup/mariadb
BACKUP_FILENAME="daily-mariadb-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="50 2 * * *"
BACKUP_PRUNING_PREFIX="daily-mariadb-backup-"
BACKUP_RETENTION_DAYS="7"
EXEC_LABEL=mariadb

BACKUP_SOURCES=/backup/mariadb
BACKUP_FILENAME="monthly-mariadb-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="50 6 1 * *"
BACKUP_PRUNING_PREFIX="monthly-mariadb-backup-"
BACKUP_RETENTION_DAYS="365"
EXEC_LABEL=mariadb

BACKUP_SOURCES=/backup/mariadb
BACKUP_FILENAME="weekely-mariadb-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="50 4 * * *"
BACKUP_PRUNING_PREFIX="weekely-mariadb-backup-"
BACKUP_RETENTION_DAYS="31"
EXEC_LABEL=mariadb

[m90]

Do you also use an influx (and so on) label or is mariadb the only label you are using? I am not entirely sure what's going on in your case but the feature works as intended, see the test case here: https://github.com/offen/docker-volume-backup/tree/26c8ba971f9e2a19f18485ae75c2776b20c3ad1c/test/commands - maybe this gives you a starting point to work off.

[alexander-zimmermann]

No, I have currently two labels. One for mariadb and one for influx.

I'm aware of your example, but this is not 1:1 compare with my setup since mine is unfortunately more complex. Things that are different are:

• you mount only one volume. no problems with the mapping can occur
• you set the environment variable in the docker compose, I need to set them in docker-volume-backup/config/
• I have two EXEC_LABEL (one mariadb and one for influx), not only one
• your EXEC_LABEL is static and set while launching docker compose up. As you suggested I set EXEC_LABEL in docker-volume-backup/config/, but that means the the variable is changed for each cron job

Just some rough ideas, where my setup can be broken.

Do we have the possibility to further increase the debug level? I would like to check if the environment variables are correct set per cron job. I can also share my entire docker-compose if you like

[m90]

Do we have the possibility to further increase the debug level?

There's no option to increase verbosity or something no.

[alexander-zimmermann]

Convinced that we have a bug ;-)

alexander@ubuntu:~/home-automation$ docker exec -it docker-volume-backup /bin/sh
~ # env
HOSTNAME=fe00f6523647
SHLVL=1
HOME=/root
AWS_S3_BUCKET_NAME=docker-backup
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=de_DE.UTF-8
AWS_ACCESS_KEY_ID=docker-backup
AWS_SECRET_ACCESS_KEY=XXX
AWS_ENDPOINT=storage.XXX
PWD=/root
TZ=Europe/Brussels
NOTIFICATION_URLS=pushover://shoutrrr:XXX

~ # cat /etc/dockervolumebackup/conf.d/gollum_daily.env
BACKUP_SOURCES=/backup/gollum
BACKUP_FILENAME="daily-gollum-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="0 2 * * *"
BACKUP_PRUNING_PREFIX="daily-gollum-backup-"
BACKUP_RETENTION_DAYS="7"

~ # set -a; source /etc/dockervolumebackup/conf.d/gollum_daily.env; set +a && backup
time="2022-07-15T10:55:54Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'mysqldump -p XXX --all-databases > /config/backup.sql' for container mariadb"
time="2022-07-15T10:55:54Z" level=info msg="Running docker-volume-backup.archive-pre command /bin/sh -c 'influx backup --compression gzip /var/lib/influxdb2/backup' for container influx"
time="2022-07-15T10:55:54Z" level=error msg="Fatal error running backup: withLabeledCommands: archive: error running pre commands: runLabeledCommands: error from errgroup: runLabeledCommands: error executing command: exec: running command exited 1"

~ # export EXEC_LABEL=Tomorrowland
~ # env
BACKUP_PRUNING_PREFIX=daily-gollum-backup-
HOSTNAME=fe00f6523647
SHLVL=1
BACKUP_CRON_EXPRESSION=0 2 * * *
HOME=/root
AWS_S3_BUCKET_NAME=docker-backup
EXEC_LABEL=Tomorrowland
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BACKUP_FILENAME=daily-gollum-backup-%Y-%m-%dT%H-%M-%S.tar.gz
LANG=de_DE.UTF-8
AWS_ACCESS_KEY_ID=docker-backup
AWS_SECRET_ACCESS_KEY=XXX
AWS_ENDPOINT=storage.XXX
BACKUP_SOURCES=/backup/gollum
PWD=/root
BACKUP_RETENTION_DAYS=7
TZ=Europe/Brussels
NOTIFICATION_URLS=pushover://shoutrrr:XXX

~ # backup
time="2022-07-15T11:01:21Z" level=info msg="Created backup of `/backup/gollum` at `/tmp/daily-gollum-backup-2022-07-15T11-01-21.tar.gz`."
time="2022-07-15T11:01:21Z" level=info msg="Uploaded a copy of backup `/tmp/daily-gollum-backup-2022-07-15T11-01-21.tar.gz` to bucket `docker-backup`."
time="2022-07-15T11:01:21Z" level=info msg="None of 2 existing remote backup(s) were pruned."
time="2022-07-15T11:01:21Z" level=info msg="Removed tar file `/tmp/daily-gollum-backup-2022-07-15T11-01-21.tar.gz`."
time="2022-07-15T11:01:21Z" level=info msg="Finished running backup tasks."

[m90]

Again, your example shows expected behavior. The first invocation does not set an EXEC_LABEL so all containers are considered. The second invocation does set an EXEC_LABEL (which apparently is not used at all), thus no containers to run a command are selected. This is exactly the way it's designed to work.

If you still think this is a bug, please extend the matching test case in this repo to fail when it shouldn't and I will have a look, otherwise I can't spend too much time on this issue here anymore. Thanks for your understanding.

[alexander-zimmermann]

Hey @m90, Ok now was able now to fix the config.

From the #129 (comment) it was not clear to me that is not enough to include the EXEC_LABEL command into the corresponding env file since *.env files without EXEC_LABEL just executed everything. In other words, each env file needs to have his own EXEC_LABEL even if no docker-volume-backup.archive-pre commands are need.

An example for other user if they have the same difficulties to understand ;-)

  # --------------------------------------------------------------------------
  # Mariadb
  # --------------------------------------------------------------------------
  mariadb:
    image: linuxserver/mariadb
    container_name: mariadb
    restart: unless-stopped
    ports:
      - 3306:3306
    volumes:
      - mariadb_data:/config
    environment:
      PUID: 1000
      PGID: 1000
      MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
    labels:
      - docker-volume-backup.archive-pre=/bin/sh -c 'mysqldump -p $MYSQL_ROOT_PASSWORD --all-databases > /config/backup.sql'
      - docker-volume-backup.archive-post=/bin/sh -c 'rm /config/backup.sql'
      - docker-volume-backup.exec-label=mariadb

  # --------------------------------------------------------------------------
  # Heimdall
  # --------------------------------------------------------------------------
  heimdall:
    image: linuxserver/heimdall
    container_name: heimdall
    restart: unless-stopped
    ports:
      - 79:80
      - 442:443
    volumes:
      - ./heimdall/config/php-local.ini:/config/php/php-local.ini
      - heimdall_data:/config
    environment:
      PUID: 1000
      PGID: 1000

  # --------------------------------------------------------------------------
  # Docker Volume Backup
  # --------------------------------------------------------------------------
  backup:
    image: offen/docker-volume-backup
    container_name: docker-volume-backup
    restart: unless-stopped
    volumes:
      - ./docker-volume-backup/config:/etc/dockervolumebackup/conf.d
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - mariadb_data:/backup/mariadb:ro
      - heimdall_data:/backup/heimdall:ro
    environment:
      <<: *default-environment
      AWS_ENDPOINT: storage.XXX
      AWS_S3_BUCKET_NAME: docker-backup
      AWS_ACCESS_KEY_ID: $MINIO_ACCESS_KEY
      AWS_SECRET_ACCESS_KEY: $MINIO_SECRET_KEY
      NOTIFICATION_URLS: pushover://shoutrrr:$PUSHOVER_TOKEN_DOCKER_BACKUP@$PUSHOVER_USER_KEY

The secret source is now that we have EXEC_LABEL even heimdall has no special docker-volume-backup.archive-pre command.

alexander@ubuntu:~/home-automation$ cat docker-volume-backup/config/heimdall_*
BACKUP_SOURCES=/backup/heimdall
BACKUP_FILENAME="daily-heimdall-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="20 2 * * *"
BACKUP_PRUNING_PREFIX="daily-heimdall-backup-"
BACKUP_RETENTION_DAYS="7"
EXEC_LABEL=heimdall

BACKUP_SOURCES=/backup/heimdall
BACKUP_FILENAME="monthly-heimdall-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="20 6 1 * *"
BACKUP_PRUNING_PREFIX="monthly-heimdall-backup-"
BACKUP_RETENTION_DAYS="365"
EXEC_LABEL=heimdall

BACKUP_SOURCES=/backup/heimdall
BACKUP_FILENAME="weekely-heimdall-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="20 4 * * *"
BACKUP_PRUNING_PREFIX="weekely-heimdall-backup-"
BACKUP_RETENTION_DAYS="31"
EXEC_LABEL=heimdall

alexander@ubuntu:~/home-automation$ cat docker-volume-backup/config/mariadb_*
BACKUP_SOURCES=/backup/mariadb
BACKUP_FILENAME="daily-mariadb-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="50 2 * * *"
BACKUP_PRUNING_PREFIX="daily-mariadb-backup-"
BACKUP_RETENTION_DAYS="7"
EXEC_LABEL=mariadb

BACKUP_SOURCES=/backup/mariadb
BACKUP_FILENAME="monthly-mariadb-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="50 6 1 * *"
BACKUP_PRUNING_PREFIX="monthly-mariadb-backup-"
BACKUP_RETENTION_DAYS="365"
EXEC_LABEL=mariadb

BACKUP_SOURCES=/backup/mariadb
BACKUP_FILENAME="weekely-mariadb-backup-%Y-%m-%dT%H-%M-%S.tar.gz"
BACKUP_CRON_EXPRESSION="50 4 * * *"
BACKUP_PRUNING_PREFIX="weekely-mariadb-backup-"
BACKUP_RETENTION_DAYS="31"
EXEC_LABEL=mariadb