optimization - de-optimization, fragmentation, how to use efficiently, lock down risk.

[newbie-02]

Considering the 'total fill' ( not of 'data' but of 'allocation blocks' and metadata ) described in btrfs/btrfs-todo#61 and the recovery with btrfs balance start -d sdX which took ~9 hours with full speed write ( hell on earth for flash drives ) to free ~74 GB of ~220 GB, bee until blocking of the device achieved:

• together with some file-level reflinking I provided in copying, and
• enabled compression,
• a near 1 : 6 storing effectivity, nice,
  but:
• produced / contributed to massive fragmentation,
• wasted ~74 GB of data space in ineffective filled allocation units, bad, and:
• also contributed to overfilling metadata, which
• finally shot the system in a hard to investigate hard lockdown ( no space left on device ), worst.

Consider such unwanted and propose:

• add some sort of storage efficiency helper, need not be 100% optimum, just 'care for',
• try to work in or at least propose read write block size of the device, better in 'erase block size'! for flash drives.
  ( no, I don't know how to query )
• avoid overfilling disks, accounting data, allocation units, metadata, once on goes below a threshold switch to 'simplyfying only' operations, pause or whatever, avoid to throw the system into blocked.

Naive proposals? My username isn't 'by chance' ;-)

[tlaurion]

I also had hard lockout by Metadata full consumption, since OS (qubesos) deploys btrfs with Metadata in DUP mode (consuming twice as much space for resiliency).

Also newbie here, learning the hard way but as said in other opened issue, some guidelines for mount options and partition creation are missing from doc, and QubesOS extensively using clones and snapshots is creating a lot of issues trying to deploy this, where love dedup would be more then ideal here instead (zfs) but qubesos cannot go that direction because licence issues and fedora not going that way.

So btrfs it needs to be, but guidelines in proper prevention mechanisms for that use case are needed, otherwise infinite iowait and last Metadata being fully filled locking me out gave me a cold shower testing this outside of real life scenario to push wyng-backup and qu esos best practices forward and having bees deployed optionally at qubesos install on btrfs deployment.

I second comments of op. Stalled attempts at #283 and tasket/wyng-backup#211