Storage: Add support for storage bucket backup (from Incus)

[boltmark]

This PR adds support for storage bucket backups. It includes cherry-picks from lxc/incus#365, and the below description comes in part from the associated incus PR.

Description

API

The following endpoints are added, along with the storage_bucket_backup extension:

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups

POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

DELETE /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>/export

Export

CLI

$ lxc storage bucket export
Description:
  Export storage buckets as tarball.

Usage:
  lxc storage bucket export [<remote>:]<pool> <bucket> [<path>] [flags]

Examples:
  lxc storage bucket default b1
      Download a backup tarball of the b1 storage bucket.

Flags:
      --compression   Define a compression algorithm: for backup or none
      --target        Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ lxc storage bucket export default bucket0
Backup exported successfully!                      

Archive

The export command creates a tarball from the defined bucket, which is structured as follows:

bucket0.tar.gz
├── backup
│   └── index.yaml
│   └── bucket
    │   └── file1
    │   └── file2

• The bucket directory contains the actual data
• backup.yml contains the bucket metadata and keys:

name: bucket0
backend: dir
pool: default
config:
  bucket:
    config: {}
    description: ""
    name: bucket0
    s3_url: ""
    location: none
  bucket_keys:
  - description: Admin user
    role: admin
    access-key: OO9O3OXLJWMED0W8P2I5
    secret-key: Ny4W9glDRFks7DzxlPQibREOAUSc1De9Yv0mOkuY
    name: admin
  - description: ""
    role: read-only
    access-key: JIY5PI6RPQWP7CAKJN57
    secret-key: APC95jzb9u7XUKw7J7AWAdVnaNVbjqj5dOLddkhT
    name: reader

Import

CLI

$ lxc storage bucket import
Description:
  Import backups of storage buckets.

Usage:
  lxc storage bucket import [<remote>:]<pool> <backup file> [<bucket>] [flags]

Examples:
  lxc storage bucket import default backup0.tar.gz
                Create a new storage bucket using backup0.tar.gz as the source.

Flags:
      --target   Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ lxc storage bucket import default backup.tar.gz bucket0
Backup exported successfully!                      

Design decisions of note

• One of the existing bucket keys is used by default, as everyone can at least read the bucket.
• A TransferManager struct is implemented, which utilizes MinIO to handle downloading files from a bucket to create the backup, and uploading files to a bucket when creating a bucket from a backup.

Overall, what's been changed

• The command storage bucket export and storage bucket import has been added to the command line interface (CLI).
• New APIs added
• Added new functionality for performing storage bucket backups and cleaning up expired backups
• Added new functionality for importing storage bucket backups
• Updated API and CLI documentation

[github-actions]

Heads up @mionaalex - the "Documentation" label was applied to this issue.

[tomponline]

please can you rebase

[tomponline]

Would you mind landing this change and the associated cherry-pick for the function in a separate PR to isolate the new functionality only in this PR. Thanks

[tomponline]

we should use api.StatusErrorf() here like we do in CreateStoragePoolBucketKey then we wont need the new ErrAlreadyDefined var.

 api.StatusErrorf(http.StatusConflict, "A bucket key using that access key already exists on this server")

Then we can check it in the call site using:

api.StatusErrorCheck(err, http.StatusConflict)

[tomponline]

we dont use logger.*f functions, but we can probably just remove these right?

[tomponline]

please can you rebase