Auto-remove documentation unclear and not working

[icebreaker-ch]

In the Settings->Auto-remove tab it is not clear how the options are logically connected: Does that mean for example
a) Remove backup IF it is older than x AND free space is less than y AND free nodes is less than z
or
b) Remove backup IF it is older than x OR free space is less than y OR free nodes is less than z

The documentation does not say anything about that.
Even though, I have backups older than one year and they are not Auto-removed.
It is also unclear which event triggers an Auto-Remove check or Auto-Remove.

I use version 1.2.1 on a Linux Mint system.

Maybe it is the same issue as described in #374 but this is almost 5 years old.

[daveTheOldCoder]

I had to edit the code and add some "logger.debug" calls to figure out what's going on.
#1094

[ceperman]

Ditto the Smart remove options - equally confusing. I would like to use them but I've not had the courage yet because I'm not sure what will happen to the existing snapshots.

However, having thought more about it, what makes sense to me is if they are conditions for retaining a snapshot, and are OR'd together. They are intended to provide decreasingly granular control as time increases, and are most clear when the time periods increase.
So if I were to set:

Keep all snapshots for the last = 14 days
Keep one snapshot per day for the last = 28 days
Keep one snapshot per week for the last = 8 weeks
Keep one snapshot per month for the last = 4 months

(plus keep one snapshot per year for all years, I have no control over this)

this would mean, for each snapshot:

• if it was taken in the last 14 days, keep it
• otherwise if it was taken in the previous 14 days, and there isn't already another in this period, keep it
• otherwise if it was taken in the previous 4 weeks, and there isn't already another in this period, keep it
• otherwise if it was taken in the previous 2 months, and there isn't already another in this period, keep it
• otherwise if there isn't already one for this year, keep it
• otherwise remove it

If the time periods don't increase successively, it still works even though there appear to be contradictions. So, for example, if I say:

Keep all snapshots for the last = 28 days
Keep one snapshot per day for the last = 7 days
Keep one snapshot per week for the last = 2 weeks, etc

as long as they are conditions for retention and are OR'd then there are no contradictions even though they don't make much sense.

The other options on the Auto-remove page look to be conditions for removal rather than retention, and so could cause conflicts if selected with Smart remove. For example, if I select "Older than 2 weeks" and Smart remove "Keep all snapshots for the last 28 days" what happens?

Can anyone confirm any of this?

[ceperman]

I revisited this subject recently and in the absence of any clarification I took a look at the code. The smart remove options are indeed criteria for retention, and are OR'd together. Overall, this is what happens ref. the Auto-remove options, and in this order:

1. If "Older than" is enabled, any snapshots meeting this criterion are deleted.
2. If "Smart remove" is enabled, then any snapshot meeting any of the "keep" criteria is retained. All others are deleted. It doesn't matter if a snapshot matches more than one of the criteria - it can't be "over-retained".
3. If "free space is less than" is enabled, the oldest snapshots are deleted until the required free space is achieved.
4. If "free inodes is less than" is enabled, the oldest snapshots are deleted until the required free inodes are achieved.

If the "Dont remove named snapshots" is enabled, named snapshots will never be deleted.

The smart remove options are easily understood when taken by themselves, but it's harder to understand how they combine to achieve what you actually want.

Say, for example, you configure a weekly backup, and want to keep all of these for a month, then just one of these per month for the rest of the year, then just one a year after that.

This is fairly easy; all you need is:

Keep one per week for the last = 4 weeks
Keep one per month for the last = 12 months
(One per year is automatic)

but you do have to configure all the keep options, so, remembering that these are criteria for retention and you've already ensured the ones you want are retained, what you set for the other options doesn't matter as long as they don't expand the criteria you've already specified (otherwise you may end up retaining more than you intended) . Say:

Keep all snapshots for the last = 1 day (or anything not greater than 28 days)
Keep one per day for the last = 1 day (or anything not greater than 28 days)

You could probably achieve the same by a different combination of values.

[aryoda]

@ceperman Thanks for digging that deep (into the code) 😄

The current dialog for this is indeed confusing:

Could you (and other users) propose changes to make this dialog more intuitive?

[ceperman]

I think the confusion with this dialog is that auto-remove is a combination of rules for deletion and rules for retention, and it's a matter of understand how these interact. It's relatively complex, and good documentation is the best answer and this is lacking at the moment.

But I would suggest making the smart remove conditions optional, so that you're not forced to use unnecessary and overlapping conditions that just confuse. This wouldn't have too serious an impact on the code I think. Some text within the smart remove box might help, such as "Snapshots matching any of the configured rules will be kept. All others will be removed".

Even though the free space and free inodes deletion take place after smart remove (code-wise), it's best to leave them where they are in the dialog, so you keep the "rules for deletions" together. If necessary the docs can explain that these aren't actioned until after smart remove because that process may free up space making further deletions unnecessary.

An enhancement could be a dummy run option for smart remove, which would report on which snapshots would be retained or removed without actually doing it. Some users might like this before committing to the real thing.

[aryoda]

An enhancement could be a dummy run option for smart remove, which would report on which snapshots would be retained or removed without actually doing it. Some users might like this before committing to the real thing.

A "test run auto-remove " button would be great!

I am afraid without existing snapshots nothing will happen during the test run and once snapshots exist they are "smartly removed" when BiT starts so a "test run" would be too late.

But if the user changes the settings the effects could be shown before clicking on "OK" via a "test run".

[daveTheOldCoder]

I am afraid without existing snapshots nothing will happen during the test run and once snapshots exist they are "smartly removed" when BiT starts so a "test run" would be too late.

I wonder how difficult it would be to have a "simulation mode". A fake set of files could be created initially, and a simulation would show the results of running BackInTime for several years.

[ceperman]

I am afraid without existing snapshots nothing will happen during the test run and once snapshots exist they are "smartly removed" when BiT starts so a "test run" would be too late.

But not if smart remove has not yet been enabled. This is an option for the user who's not yet had the courage to enable it. :-)

[buhtz]

Please see #1679 also related to this discussion.
I will carefully review your comments and try to find a solution. I think it is a good idea to separate between re-design the GUI and re-design the behavior.

[buhtz]

Might not be directly relevant to this issue. But I investigated the current auto-/smart-remove behavior and documented it. I found inconsistent behavior and minor bugs. There is a lot potential to improve and refactoring.

Before starting with this I would like to request your opinions and suggestions about the next steps. Please see the issue #1945 for details and a mockup for the new auto-remove dialog-tab.

Feedback and suggestions welcome. We appreciate your feedback.