Add possibility to download IPFS images

[afbjorklund]

See https://docs.ipfs.tech/how-to/kubo-basic-cli/ for ipfs

https://docs.ipfs.tech/how-to/address-ipfs-on-web/#native-urls

ipfs://{cidv1}/path/to/resource (see also https://cid.ipfs.tech/)

Closes #2407

[afbjorklund]

The lima.yaml would then look something like:

- location: "ipfs://QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"

(the file name is only used for decompression)

Note that the CID digest is not the file digest:

https://cid.ipfs.tech/#QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK
6278B63498EB92816C50A53202EE3CBEE6FC0F92F97B97CB0AB0A4AE65CCBE38

https://docs.ipfs.tech/concepts/content-addressing/#cids-are-not-file-hashes

(small detail: zQmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK in multihash format is the same as
"sha256:6278b63498eb92816c50a53202ee3cbee6fc0f92f97b97cb0ab0a4ae65ccbe38" in text format)

[afbjorklund]

Note: the ipfs tool will output v0 by default, unless using --cid-version 1

v0: QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK
v1: bafybeig5sch22ecfox7gq724rz7uivydwvnnpuqdcnjz72iwelgtrakzui

$ ipfs add --cid-version 0 ubuntu-24.04-server-cloudimg-amd64.img 
added QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK ubuntu-24.04-server-cloudimg-amd64.img
 453.00 MiB / 453.00 MiB [=================================================================================================================] 100.00%
$ ipfs add --cid-version 1 ubuntu-24.04-server-cloudimg-amd64.img 
added bafybeig5sch22ecfox7gq724rz7uivydwvnnpuqdcnjz72iwelgtrakzui ubuntu-24.04-server-cloudimg-amd64.img
 453.00 MiB / 453.00 MiB [=================================================================================================================] 100.00%

The CID version doesn't really matter to Lima, but CIDv0 is deprecated.

https://docs.ipfs.tech/concepts/content-addressing/#cid-versions

[afbjorklund]

Using ipfs cat to get our own progress, and ipfs ls to calculate the size.

Hash                                                        Size     Name
bafybeibnerap2c5tnmqvyftmyxftftjewvo52acwv2gg6thpipyw5zx7fe 45613056 
bafybeidrbml5blbqj2i67x5gujqbwluis6jy5daiso6ir5pakx4hseztum 45613056 
bafybeie56ruaziz43jj3e5iih5r3fyzudeurqbpts6vwigf7nbreoep5de 45613056 
bafybeifva7p55d4szahrrxqjdjqqqlcrcfxm7n2vnpmwucrdkvcexxngmu 45613056 
bafybeie5wduudyjkbcyjvcif7i6gliipapvl5cw4wia3fqoanxunud7dyi 45613056 
bafybeidezz5b2zng3etcwbn4vnu3tn4ao3wrh5qdwzoyo7ywdp7rbymgzq 45613056 
bafybeiaf7vcteys366fmjiclktnzwuqpygpn4u2ftqrl62juaq5sgjw4be 45613056 
bafybeiasvhz6mljylbqwygb7jvp6d6sohaw325yuan5awlo7shjrlslb7m 45613056 
bafybeieiuxw2i5nkelq5tihd2tgmtxjst2ln6rslt35jxslcz4w4mjsnha 45613056 
bafybeiest5wjnmjwyp7vgawfbityfnkgqpr27bfleysbhdrhldcwxw4yq4 45613056 
bafybeif7btzsv7657kfxu7qte6sh6zgpvpsrj7dnou7usj2nwv4zf77ft4 18874368 

So now IPFS address looks the same as HTTP address, with "description":

Downloading the image (ubuntu-24.04-server-cloudimg-amd64.img)
453.00 MiB / 453.00 MiB [---------------------------------] 100.00% 336.20 MiB/s

Instead of the output that you get from ipfs get, that also could change.

Saving file(s) to /home/anders/.cache/lima/download/by-url-sha256/1855c5dccbd6db83ea6c81c276e0440ad9f156584e2ced824290186f1dae563b/data
 453.00 MiB / 453.00 MiB [==================================================================================] 100.00% 0s

[afbjorklund]

At first I was thinking that calculating the digest was unnecessary, since it already has one included in the storage.

But we still want to compare the download with the digest we are expecting, to make sure it's the same image...

[AkihiroSuda]

Looks good, but needs a documentation

[afbjorklund]

There is a design flaw with this approach. Currently it would look like:

- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"
- location: "ipfs://QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"

That means the checksum is duplicated, between the transports. Maybe:

- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20240423/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3"
  cid: QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK

---

Note:

The CID can be calculated, without adding the image to the disk store:

$ sha256sum ubuntu-24.04-server-cloudimg-amd64.img 
32a9d30d18803da72f5936cf2b7b9efcb4d0bb63c67933f17e3bdfd1751de3f3  ubuntu-24.04-server-cloudimg-amd64.img
$ ipfs add --only-hash --quieter ubuntu-24.04-server-cloudimg-amd64.img 
QmUy3RRqbpsxXYQ7yp4h2koFHdGdVTCfRiqCBLkw1JobUK

[afbjorklund]

Unfortunately, Content-Type and Last-Modified are not provided by IPFS...

Current gateway only use heuristics like file magic and relative freshness.

Note: new implementations are supposed to use CID version 1 (not version 0):

$ ipfs add --cid-version=1 --only-hash --quieter ubuntu-24.04-server-cloudimg-amd64.img 
bafybeig5sch22ecfox7gq724rz7uivydwvnnpuqdcnjz72iwelgtrakzui