Impl. backup rotation scheme keeping at least 1 copy of all files

[getreu]

I am trying to implement a backup rotation scheme with Borgbackup2. My concern is, how to
guarantee, that I always keep at least one copy of all my files in the repository. This is my first try:

CHOOSEFILES |
borgbackup2 create --paths-from-stdin --paths-delimiter "\0" --verbose --stats ::'{hostname}-{now}'   

borgbackup2 prune  --list --match-archives '{hostname}-*' --show-rc \
    --keep-daily    7 --keep-weekly   4 --keep-monthly  6      

borgbackup2 compact

My (beginner) mental model about what happens:

1. Archives contain pointers to backuped content in the repository. When prune deletes an archive, it does not delete any content, it only deletes a set of pointers.
2. create checks for all incoming files, if the file is already present in the repository. If yes, just a pointer to that file is stored in the archive. In any case, all incoming file is represented by a pointer in the archive's pointer set. Even those files who have been present in the repository already before this backup and no content has been transferred.
3. compact acts as "garbage collector" deleting all content that is not referenced by any archive (pointer set).

Is my understanding right?

If yes, is it guaranteed, that even in case I only keep the last archive: do I still posses a copy of all my files in this last archive?

[ThomasWaldmann]

borg2 is still in beta testing, so I assume this is not your only backup.

A1: if you use delete or prune, borg logically deletes chunks if their refcount reaches 0, so it is not just removing the references. this might change in future though, see the breaking tickets.

A2: presence checks are done for chunks, not for files. of course, small files might be just 1 chunk.

A3: compact physically removes the already deleted chunks (see A1) from the segment files. it takes an old segment file with such "holes" and writes all non-deleted chunks to a new, compact, smaller segment file.

If you keep some archive, you will have all files that you put into it at the time you created that archive. Logically, every borg archive is a "full" backup.

[ThomasWaldmann]

BTW, there is usually not much motivation to only keep the last archive, but usually one rather keeps 10..100 archives. These often do not need much more space than a single archive.

Of course that depends on the deduplication efficiency and that depends on how much data actually changed between these backups.

[getreu]

BTW, there is usually not much motivation to only keep the last archive

Sure, this was a theoretical example to assert: Logically, every borg archive is a "full" backup.

borg2 is still in beta testing, so I assume this is not your only backup

Life is about taking risks. And, I am confident that you guys will publish v2 soon (and maybe add more features at
some moment in the future?). ;)

A1: if you use delete or prune, borg logically deletes chunks if their refcount reaches 0, so it is not just removing the
references. this might change in future though, see the breaking tickets.
A2: presence checks are done for chunks, not for files. of course, small files might be just 1 chunk.
A3: compact physically removes the already deleted chunks (see A1) from the segment files. it takes an old
segment file with such "holes" and writes all non-deleted chunks to a new, compact, smaller segment file

refcount = 0 sounds like a garbage collector, it makes sense to me.

If you keep some archive, you will have all files that you put into it at the time you created that archive. Logically, every borg archive is a "full" backup.

Thank you. I did not find this (for me important) confirmation in the docs!

Another question: Does this last archive rule guarantee, that even the clumsiest prune command will always leave at least the latest archive in the repository?

[ThomasWaldmann]

borg2 might still take a while, as I want to try some rather big changes still (not sure about the outcome yet).

Not sure what you mean by "last archive rule". When setting up prune rules, always use --dry-run --list at first.

[getreu]

I do not really understand the excerpt below. Is this some security rule to insure that prune does not empty the entire repository?

As of borg 1.2.0, borg will retain the oldest archive if any of the secondly, minutely, hourly, daily, weekly, monthly, or yearly rules was not otherwise able to meet its retention target. This enables the first chronological archive to continue aging until it is replaced by a newer archive that meets the retention criteria.

[ThomasWaldmann]

That means e.g. if you say "keep 10 yearly archives" and you only have 2 yet (so the target of 10 is not yet met), it would also keep another one (the very oldest) even if it does not qualify as "yearly".

E.g.:

• 2023-12-31
• 2022-12-31
• 2022-02-03 (oldest archive present)

[getreu]

Ok, I understand. Thank you.