multiple sources for multiple hosts

[braham2019]

Currently, I'm using a generic bash script that, depending on the host, has different backup options, such as: backup /etc, /home, docker or mysql. One script to rule them all ;-)

I'm new to restic and resticprofiles, and am charmed by it. And I have done a few tests, but I can't seem to get my head around the way to implement this in a single profile yaml file.

I can call a group, to go through certain backups, such as backing up crontab through stdin, backing up sql trough sdtin, or regular file backups. So it will iterate through the list to backup just fine.

So I have two main challenges:

1. I call the backup script on a server and dump it all in a SFTP repository. works fine. but... for certain hosts, I want to backup to a local repository. So depending on the host, I have a different repository.
2. how to use variables for SQL backups. if I have two mysql hosts, I have two different sets of credentials to pass through the stdin command.

I know I can create a different yaml file for every single host I have, but then I struggle with keeping the backups and rules consistent on all hosts. I prefer to have a single script for all hosts and all types of backups.
I know I can use mixins, but fail to see a way to use this based on the hostname of the server to backup.

Any ideas or suggestions ? thx

[creativeprojects]

Hi!

Unfortunately there's no one size fits all configuration. But since we got a few similar questions recently I think we should provide a few use cases in the documentation.

My own configurations are generated for each server from an ansible template. This is something I could document though.

In the meantime have you find any interesting information from the other discussion #275?

[jkellerer]

@creativeprojects , maybe some sort of conditionals for the activation of a profile would be nice to make it easier to create reusable config files?

[creativeprojects]

You mean, in this particular case, you're thinking about something like this?

remote_profile:
  as: profile
  when: $HOSTNAME == "remote"

local_profile:
  as: profile
  when: $HOSTNAME == "local"

[jkellerer]

Something like this, wasn't thinking of aliases but that would also work with base profiles. Just worried that it makes things a bit complicated in the end. Also I wasn't particularly thinking about this case but being able to enable or disable profiles based on evaluations might help in general.

[jkellerer]

As a possible answer to your 2 main challenges:

1. I'd probably use 2 different base profiles and select them dynamically e.g. based on hostname
2. For SQL backups, I'd probably use a password file rather than adding credentials directly to the profile.

E.g.:

# yaml-language-server: $schema=https://creativeprojects.github.io/resticprofile/jsonschema/config-2.json
version: "2"

##
# {{ $HostsWithLocalBase := list "hosta" "hostb" "hostc" }}
##
# {{ $Hostname := printf "/%s/" ( .Hostname | lower ) }}
# {{ $IsLocalBase := printf "/%s/" ( $HostsWithLocalBase | join "/" | lower ) | contains $Hostname }}
# {{ $Base := "base-sftp" }}
# {{ if $IsLocalBase }}{{ $Base = "base-local" }}{{ end }}
##
# Hosts with base-local: [ {{ $HostsWithLocalBase | join ", " }} ]
# Host {{ $Hostname }} uses {{ $Base }}
##

profiles:
  # defaults for all
  default:     
    initialize: true

  base-sftp:
    inherit: "default"
    repository: "sftp://..."
    password-file: ""

  base-local:
    inherit: "default"
    repository: "file:/path/to/repo"
    password-file: ""


  backup-xyz:
    inherit: "{{ $Base }}"

    backup:
      source: ...

  backup-mysql:
    inherit: "{{ $Base }}"

    backup:
      stdin-command: "cat /etc/mysql/backup-password.txt | mysqldump --all-databases --order-by-primary --password"
      stdin-filename: "dump.sql"

Resulting in:

resticprofile --config profiles.yaml --trace show

2023/12/22 12:29:19 Resulting configuration for profile 'default' ('profiles.yaml' / replace=true):
====================
  1: # yaml-language-server: $schema=https://creativeprojects.github.io/resticprofile/jsonschema/config-2.json
  2: version: "2"
  3: 
  4: ##
  5: # 
  6: ##
  7: # 
  8: # 
  9: ##
 10: # Hosts with base-local: [ hosta, hostb, hostc ]
 11: # Host /hostb/ uses base-local
 12: ##
 13: 
 14: profiles:
 15:   # defaults for all
 16:   default:     
 17:     initialize: true
 18: 
 19:   base-sftp:
 20:     inherit: "default"
 21:     repository: "sftp://..."
 22:     password-file: ""
 23: 
 24:   base-local:
 25:     inherit: "default"
 26:     repository: "file:/path/to/repo"
 27:     password-file: ""
 28: 
 29: 
 30:   backup-xyz:
 31:     inherit: "base-local"
 32: 
 33:     backup:
 34:       source: ...
 35: 
 36:   backup-mysql:
 37:     inherit: "base-local"
 38: 
 39:     backup:
 40:       stdin-command: "cat /etc/mysql/backup-password.txt | mysqldump --all-databases --order-by-primary --password"
 41:       stdin-filename: "dump.sql"
 42: 
====================

References:

• configuration/inheritance
• configuration/templates
• configuration/variables

[braham2019]

Thanks, I will give it a go when I have some time available. For now I have redone my batch file to use restic instead of rsync using my faithful batch file and command line parameters. I'm not a programmer (just a hobbyist with a few servers that need backing up), so it always takes a while for me to digest new stuff.