Dealing with attic issues

[maltefiala]

Here is a list of all open issues in attic, acquired with this dirty python script . I suggest we go through all of them and tick them when fixed.

• init: missing parameter to avoid error if repository already initialized
• separable repository format
• ssh:// after network re-connection and SIGINT issues
• Suggestion: Allow multiple servers to back up to the same repo at the same time
• Add metadata search index for more responsive FUSE
• Force-include the specified paths in 'create'

Done / tracked here / invalid / out-of-scope:

• Update installation instruction in README
• Cannot build with pip because of EVP_CIPHER_CTX error
• transaction recovery issue?
• Attic stores hard link to inaccessible file
• attic mount doesn't work
• build on fedora
• safe to interrupt backup?
• Has anyone got working on windows
• Cannot build with pip because of EVP_CIPHER_CTX error
• Attic stores hard link to inaccessible file
• File search Request
• Allow 'list' while backup is in progress
• --include switch
• Expose External Crypto providers
• New security changes broke remote access to repos
• Incorrect directory permissions during extract
• where shall parameters get stored?
• Flexibility and cleanup
• "extract" command has no destination switch?
• Has anyone got working on windows
• Repository clone command?
• Parallel backups from different hosts crash attic
• Cannot mount due to llfuse error
• ZFS snapshot ingestion support
• Let attic know it should wait before beginning
• aes-gcm support
• aes-gcm or aes-ocb?
• exclusive=x flag and RemoteRepository
• "Initializing cache..." does not save progress
• I/O Error on CIFS Mounts?
• undefined symbol: EVP_aes_256_ctr error when running attic
• draw a picture from the internals documentation
• How to list files corresponding to corrupted data segments?
• attic-mount misses files that can be extracted through attic-extract
• PermissionError
• setup.py: seek /opt/pkg (pkgsrc OS X) for openssl
• Unable to download precompiled binaries
• Use environment variable to set the tmp directory
• msgpack.exceptions.UnpackValueError
• attic check with ATTIC_PASSPHRASE -- getpass.py/terminal issue
• non-zero exit status -11
• SSH and Ctrl-C leave attic serve instances alive and prevent further upload
• Typo: fileystem -> filesystem
• Parity check/recover considered?
• Can't subscribe to mailing list
• Installation problems on CentOS 6.7
• Locally backup S3 buckets
• Tests failing on Gentoo Linux
• Debian Jessie installation instructions for pip3
• acl_use_local_uid_gid has suspicious code
• Cache location
• Data integrity error after running out of disk space
• OverflowError when attempting to create a backup
• attic crashed with "malloc failed"
• ValueError on attic check --repair
• Error with UnicodeDecode
• Conversion from rdiff-backup
• // , Installation fails on CEntOS 7.1.
• // , gpg key for attic-backup.org
• Default repository
• low-impact - lrucache code appears to leak fds without closing them
• Optimization: asynchronous writeback
• repository: two minor improvements
• Tiny fixes: printf error messages and gitignore
• attic remote backup ends in broken pipe
• attic/_version.py from github releases change over time
• Attic segfaults
• Let attic take a remote path option for cases where attic cannot be installed in standard places (like rsync, git, etc)
• Honor nodump flag
• osx pip3 install Attic - command not found
• Allow to ignore file by size
• Make the script for automatic backup finding the hostname
• --include feature
• OverflowError: value too large to convert to int
• Undocumented standalone binaries
• Undocumented environmente variables
• Restore remote data inplace with deduplication or rsync algorithm
• Add --files-from and --files-from0 options
• create flag to print backed up files
• "Failed to acquire write lock" FUSE
• int32 chunk reference counter, is it a problem?
• merge multiple repos into one
• Exit gracefully when output pipe is broken
• Performance: attic create stalls while index is read back
• typo in docs theme css
• st_mtime_ns precision or rounding related test failure
• memory usage is too high
• Question: list files modified/added in archive
• Exception: hashindex_set failed (malloc failed)
• Exception raised during create
• Trailing slash on "init" confuses attic
• Add disk space preview command
• speed / compressed tradeoff parameter?
• Doc symlinks are not followed on attic create.
• sigusr1 / sigint signal handler to print current file infos
• implement --timestamp, iso8601-like utc timestamp string or reference file
• simple sparse file support, fixes #256
• crc32 computation partly redundant?
• use posix_fadvise to avoid spoiling the OS cache
• Documentation for 'attic serve'
• attic serve is not documented on attic-backup.org
• Question about subsequent backups, and about pruning
• improve and simplify key modes
• potential counter/IV uniqueness issue
• use poly1305-AES for faster MACs
• attic create -n/--dry-run
• ACLs do not work when using attic mount
• Data integrity error due to index mismatch for key
• Specify the date and time of the backup
• Remove a path prefix on backup and/or restore
• Specify the "extract to" location
• improve restores by posix_fallocate
• attic does not backup UNIX domain sockets
• upgrade versioneer so it is pep440 compliant
• handle sparse files
• Adds a note about the ACCDCC2D signing key to the installation documentation
• Fix test setup for 32bit platforms, partial fix for #196
• wrong code in /bin/attic script
• make system more responsive by using the fadvise DONTNEED
• implement attic rename repo::oldname newname
• implemented create --progress
• own lrucache?
• catch all Exceptions, transmit traceback back from remote
• fix some exception handlers
• fix noatime mode, fixes #243
• atime preservation code is broken
• Refactor 'list' and 'prune' operations on remote repositories to run on the server side
• fix misleading hint the fuse ImportError handler gave, fixes #237
• misleading error msg when importing fuse support fails
• give specific path to xattr.is_enabled(), disable symlink setattr call ...
• Traceback when running 'check --repair' on a repository
• check --last N
• avoid defect python-msgpack releases, fixes #171, fixes #185
• check unpacked data from RPC for tuple type and correct length, fixes #127
• improve on ready-only targets, see issue #202
• Delete repo
• better attic create -v output
• add global option --no-cache-files to lower memory consumption
• fix traceback when trying to do unsupported passphrase change, fixes #189
• datetime does not like the year 10.000, fixes issue #139
• Implement mid-file checkpoints?
• improve exception handling / diagnosis
• helping out with merge reviews and issue triage
• compatibility policy / support timeframe
• cleanup crypto.pyx, make it easier to adapt to other modes
• OpenSSL's EVP_* interface not used correctly?
• version number creates wrong impression
• hash performance sha256 vs. sha512
• stdin and stdout support
• Misc. Fixes / Improvements
• Error while creating backup - 'Repository' object has no attribute 'segments'
• Document source of python 3.4 libraries bundled with releases
• More user friendly message if repository is on read-only storage
• attic hangs when initializing repository on a cifs mount
• Attic mount dies (Input/output error) when backgrounded
• Win32
• Extra debug information for 'fread failed'
• attic create fails with "assert segments[segment] == 0"
• How to delete or rename a repository
• add more .c files to .gitigore
• Retry on Input/Output error
• Changing passphrase fails: AttributeError: 'PassphraseKey' object has no attribute 'change_passphrase'
• Changing/removing keyfile password?
• FUSE: reflect deduplication in allocated blocks
• attic does not respect XDG_CACHE_HOME
• Creating archive with a slash breaks mount option
• Suggestion: attic check --last N
• extended attributes in 0.14 / linux
• KeyError: 'cache'
• attic check --repository-only reports "Error reading segment ###" at different locations
• Pull mode
• Enhancement: Add archive diff command
• Only allow whitelisted RPC calls in server mode
• Some file operations not working correctly in mounted repo
• Issue with modification dates of parent directories in mounted archives
• Traffic needed for 'list' and 'prune' operations on a remote repository increases linearly with time
• Integrity error: 300k files 100GB of data.
• Restrict "serve" command to "create" operation only
• Traceback on a frequent backup
• list changed files in info
• Attic leaks file descriptors during checkpoint
• document that multiple clients can backup to a single remote
• document Internals better
• document better different dependencies
• document that hardlinks and symlinks are supported
• Normalize source/exclude paths before matching
• Suggestion: attic info --latest
• Suggestion: Add a comment string to an archive
• Suggestion: Mark a specific archive as "never prune"
• Error backing up a changing filesystem
• Implement --exclude-if-present
• Specifying date or ability to change it.
• attic list bombs with "ValueError: year is out of range"
• Mounting a entire repo is not working
• Support for Amazon S3?
• pip install on Ubuntu 14.04 fails
• FUSE 2.8.0 required?
• Document that hard links work
• Prune help unclear about what it will delete
• Attic doesn't handle connections issues when using remote repo
• --exclude-if-present
• TypeError when trying to connect to a remote repository
• Cache is newer than repository, refusing to continue
• UnicodeEncodeError
• cifs destination exception: listxattr failed
• attic freezes sometimes
• Subkeys
• Add a interface to report current progress
• Include other compressors like XZ for smaller size and LZO for speed
• The "self contained" builds are missing the llfuse dependency
• Add bandwidth limit support
• Slow backup speed (~12MB/s)
• Support custom ssh binary / ssh options?
• description of exclude wildcards
• Backup speed getting worse after cleanup
• Copy repository to secondary location
• Fixed buffer size a waste of resources/overhead?
• Speed up directory scanning by ordering by inode
• options to not compare inode numbers, and disable acl / xattr
• Re-encrypt a repository with a new key
• Convert unencrypted repositories to encrypted repositories
• Document at "Usage->Attic" that init remote need installation on remote
• create directories for mount points
• docs bounty: let's crowdfund the attic docs wrinting ( 100 usd is my personal pledge. )
• Poor FUSE performance
• Windows Support
• Attic and large files
• Documentation on how I used Attic Backup (full system backup)
• print current status/filename on SIGUSR1 in non-verbose mode
• More informative "attic create -v" output
• Document exit code(s)
• Use logging lib for messages
• Document known file cache and add an option to disable it
• Add option to disable file chunks cache to reduce memory usage
• Document how repository encryption works
• Document how deduplication works
• Document repository format
• BlockingIOError in verbose mode
• Archive stdin or fifo
• config file whis tasks

[ThomasWaldmann]

I ticked all the stuff that's currently in borg master branch. There is some more stuff already fixed/improved that is not merged into there yet.

I am also ticking the items now that I think should be closed (because duplicate, irrelevant, stale, not enough infos, ...).

I'm also ticking the stuff that has own issues in the borg issue tracker.

[ThomasWaldmann]

about jborg/attic#252 (fadvise DONTNEED): it is merged (and AFAICS using fadvise DONTNEED had positive effects).

https://github.com/borgbackup/borg/blob/master/borg/_chunker.c#L160

People seem to disagree whether that is beneficial or not.
I'll reconsider as soon as facts or at least good reasoning is available that indicates using it is wrong.

[ThomasWaldmann]

about pull mode: it is ticked because we have an issue (which is a bit more general) in our issue tracker: #36

[anarcat]

it would have been nice to clarify which tickets are resolved, in the issue tracker or ignored instead of just ticking them off here... now we don't clearly know what's what...

[ThomasWaldmann]

@anarcat I see your point, but I also don't want to make it even more work.

I mainly see the unchecked attic issues as stuff that has still to be looked at (and maybe dealt with), so that one does not have to go through all the issues there again and again. Of course we also need to deal with the issues in borg issue tracker.

What's really fixed can be seen more easily in CHANGES.rst.

[anarcat]

yeah ok, just trying to fix #224 here :)

[anarcat]

jborg/attic#99 implemented through #248

[anarcat]

jborg/attic#181 implemented through #247

[ThomasWaldmann]

After doing a large volume test myself (see #216) and not finding issues, I'll check these ones as "can't reproduce" now:

jborg/attic#264
jborg/attic#176
jborg/attic#166

[ThomasWaldmann]

Note: I put a checkmark in that (native) "windows support" attic issue. Not because we have native windows support (only cygwin), but because the related changesets there are rather messed up.

[anarcat]

jborg/attic#18 has been reopened in #315

[ThomasWaldmann]

jborg/attic#70 is 404 (huh? how can that happen?)

but as we recently improved FUSE performance, I've just checked it as solved.

[ThomasWaldmann]

i checked jborg/attic#367 in the list as we track this in #225.

[pacman88]

Hi
In the list there is a point named "Pull mode" which is ticked. That is great since it is exactly what I need (schedule and manage backups from one central server). Unfortunately I can't find any hint in the docs how to make backups in pull mode.

so does that mean it is fixed/implemented but I am missing the docs or is it ticked by mistake?
(I would love to see pull mode implemented)

[anarcat]

@pacman88 see #5 (comment) and #36

[pacman88]

@anarcat: thx - and sorry for just skimming over the thread

[ghost]

Hi,

I'm unable to run borg create as I'm running into #317 (something related to NFS). Would you like me to open a separate issue in this repo?

info:

Platform: OpenBSD orchid.home 5.9 GENERIC.MP#1854 amd64 amd64
Borg: 0.30.0 Python: CPython 3.4.4

[ThomasWaldmann]

@pyrohh it's unclear what your issue is. sure open a new ticket and provide enough information so we can reproduce.

[ghost]

@ThomasWaldmann I'll do that, and I meant to link to jborg/attic#317, not #317 in this repo.

[ThomasWaldmann]

@pyrohh ah :) but i doubt it is same issue as that one as we do not use fcntl locking any more.

[ghost]

False alarm, everything works after a reinstall... must have been the boogie man. Anyway, this is such a nice piece of software, thanks for working on it :D

[ThomasWaldmann]

jborg/attic#103 checked, we have it now in ticket #661.

[ThomasWaldmann]

checked attic 100, quite old and no response there.

[JensRantil]

FYI, just noticed jborg/attic#219 is closed.

[ThomasWaldmann]

@JensRantil yes, but in borg, we'll do aes-gcm, likely in borg 1.2.

[ThomasWaldmann]

attic 120 is borg #1462.

[ThomasWaldmann]

checked attic 227 - in PR #1371 rpc api for exclusive flag was added, so remote and local repos are now more similar in how they lock. also, the deadlock was fixed.

[ThomasWaldmann]

checked attic 383 - can be solved with borg with-lock ...

checked attic 384, out-of-scope / too special.

[ThomasWaldmann]

attic 211 and 219 checked, tracked here by #1044 and #45.

[ThomasWaldmann]

attic 131 checked. It is a rather old ticket, the tracebacks do not contain enough information.

Also, this issue is likely related to repository locking which is very different in borg (mkdir-based and since 1.0.7 immediately-exclusive) than in attic (posix locks, lock upgrades).

[ThomasWaldmann]

Checked attic 387 - borg works on cygwin and there is an ongoing windows port in branch "windows".

[ThomasWaldmann]

Checked attic 145, is locally tracked in #1695 now.

[ThomasWaldmann]

attic 276 is now tracked locally in #1751.

[ThomasWaldmann]

attic 195 is now tracked locally in #1783.

[enkore]

attic 345 has many local equivalents, including #1044 #672 #747 #45 and so on

[enkore]

attic 292 is solved since 1.0.8 (#1524).

from tw: attic 292 is about using different repo urls, borg #1524 is about saving the answer in any case, not just when doing write ops.

[enkore]

attic 210 is implemented with Manifest.config and borg_security_dir. Maybe create a internals doc ticket there?

from tw: yes that would be good, also for the changelog.

[enkore]

attic 207 is long implemented in Borg :) The crypto stuff has it's own tickets (see above).

from tw: some is implemented, some will (hopefully) come with 1.2. but yes, we can check it as we have local tickets.

[enkore]

attic 182: fuse archives are cached via a tempfile in the page cache. Resolved?

from tw: no, attic 182 is about a search index to optimize access to the right chunks.

[enkore]

attic 117 resolved in attic (also: fuse versions view, fuse repository mount, borg diff).

from tw: you mean "resolved in borg"? yes, fuse versions help, although it is not a general "file search" functionality. it only helps if you know the precise path.

[enkore]

attic 110: better locking is now done, otherwise it's covered by #768 and the like

from tw: ok, similar enough.

[enkore]

attic 104 is tracked locally in #1406

from tw: ok

[ThomasWaldmann]

attic 393 is handled by borg #2092.

[ThomasWaldmann]

attic 395 checked - borg can be built with openssl 1.1.

[vjon]

attic 386: I've opened #2415 to discuss this issue in the general case.

[ThomasWaldmann]

I'm closing this now. If we are missing something important here in the borg issue tracker, please file an issue here (after checking we do not already have one for it).