Include locked melange config in control section

[jonjohnsonjr]

This adds a .melange.yaml file to the control section of each built APK which includes the locked (resolved and apko-solved) melange configuration used to build the package.

I've fixed the one-arch.yaml example because it didn't parse properly and wanted to use it as an example because it contains a test.

$ tar -tf packages/x86_64/one-arch-0.0.1-r0.apk                                      
.PKGINFO
.melange.yaml
var
var/lib
var/lib/db
var/lib/db/sbom
var/lib/db/sbom/one-arch-0.0.1-r0.spdx.json

tar -Oxf packages/x86_64/one-arch-0.0.1-r0.apk .melange.yaml

package:
  name: one-arch
  version: 0.0.1
  epoch: 0
  description: an example of how target-architecture works
  commit: e59b1d70a24f41360748a5e308640bbae2c45c18
  target-architecture:
    - x86_64
  copyright:
    - license: Not-Applicable
  resources:
    cpumodel: host
environment:
  contents:
    repositories:
      - https://packages.wolfi.dev/os
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    packages:
      - busybox=1.37.0-r0
      - ca-certificates-bundle=20241010-r2
      - glibc=2.40-r3
      - glibc-locale-posix=2.40-r3
      - ld-linux=2.40-r3
      - libcrypt1=2.40-r3
      - libgcc=14.2.0-r5
      - libxcrypt=4.4.36-r8
      - wolfi-baselayout=20230201-r15
  accounts:
    run-as: ""
    users:
      - username: build
        uid: 1000
        gid: 1000
        shell: ""
        homedir: /home/build
    groups:
      - groupname: build
        gid: 1000
        members:
          - build
  archs:
    - amd64
  environment:
    GOMODCACHE: /var/cache/melange/gomodcache
    GOPATH: /home/build/.cache/go
    HOME: /home/build
pipeline:
  - runs: |
      echo hello package
test:
  environment:
    contents:
      repositories:
        - https://packages.wolfi.dev/os
      keyring:
        - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
      packages:
        - busybox
        - one-arch
  pipeline:
    - if: '"x86_64" == "x86_64"'
      runs: |
        echo hello test
    - if: '"x86_64" == "aarch64"'
      runs: |
        echo "BAD ARCHITECTURE"
        exit 1

diff one-arch.yaml <(tar -Oxf packages/x86_64/one-arch-0.0.1-r0.apk .melange.yaml)

diff examples/one-arch.yaml <(tar -Oxf packages/x86_64/one-arch-0.0.1-r0.apk .melange.yaml)   
5,7c5,6
<   description: "an example of how target-architecture works"
<   copyright:
<     - license: Not-Applicable
---
>   description: an example of how target-architecture works
>   commit: e59b1d70a24f41360748a5e308640bbae2c45c18
10c9,12
< 
---
>   copyright:
>     - license: Not-Applicable
>   resources:
>     cpumodel: host
18,19c20,47
<       - busybox
< 
---
>       - busybox=1.37.0-r0
>       - ca-certificates-bundle=20241010-r2
>       - glibc=2.40-r3
>       - glibc-locale-posix=2.40-r3
>       - ld-linux=2.40-r3
>       - libcrypt1=2.40-r3
>       - libgcc=14.2.0-r5
>       - libxcrypt=4.4.36-r8
>       - wolfi-baselayout=20230201-r15
>   accounts:
>     run-as: ""
>     users:
>       - username: build
>         uid: 1000
>         gid: 1000
>         shell: ""
>         homedir: /home/build
>     groups:
>       - groupname: build
>         gid: 1000
>         members:
>           - build
>   archs:
>     - amd64
>   environment:
>     GOMODCACHE: /var/cache/melange/gomodcache
>     GOPATH: /home/build/.cache/go
>     HOME: /home/build
21,22c49,50
<   - runs: echo hello package
< 
---
>   - runs: |
>       echo hello package
31a60
>         - one-arch
33c62
<     - if: ${{build.arch}} == "x86_64"
---
>     - if: '"x86_64" == "x86_64"'
36c65
<     - if: ${{build.arch}} == "aarch64"
---
>     - if: '"x86_64" == "aarch64"'

One thing worth noting is that the test images aren't resolved here because we don't resolve them at build time. I think this is fine.

There are also a bunch of fields that we could probably make more omitempty-y by switching to pointers if we wanted to make this output a little more terse, but I'm reluctant to do that as part of this change.

[smoser]

Does apk and apko somehow know to not treat .melange.json as a "normal" file?
I assumed that they somehow knew explicitly that .PKGINFO wasn't to be put onto the filesystem or considered for conflicts.

[smoser]

The other thought that I had is that this could leak data from a source yaml file that was assumed to be private.
"proper" yaml comments are stripped I'm sure, but something like this would get through:

pipeline:
  - runs: |
      # here I put the comment in the body of a 'runs', which means
      # it will make its way into the binary output
      make mushrooms
      # NB: I think that anyone that likes mushrooms or green bikesheds should go jump in a lake

[smoser]

This does mean that a re-build of a package will be not bit for bit identical binary output if any dependencies were updated.

[imjasonh]

This does mean that a re-build of a package will be not bit for bit identical binary output if any dependencies were updated.

Yes, and I think that's a feature and not a bug.

It's very likely that if a build dependency changed the resulting package would be different anyway, and this makes it easier to determine if a package is reproducible given the same exact dependency package-versions or not.

[jonjohnsonjr]

Does apk and apko somehow know to not treat .melange.json as a "normal" file?
I assumed that they somehow knew explicitly that .PKGINFO wasn't to be put onto the filesystem or considered for conflicts.

Yep! APKs have gzip splits between sections. It looks like a single tar.gz file but it's essentially 2 (or 3 with signatures) gzip streams concatenated together.

The control section contains metadata files about the package at the beginning, including .PKGINFO, which don't actually get installed by apk add.

[jonjohnsonjr]

The other thought that I had is that this could leak data from a source yaml file that was assumed to be private.

I have addressed this by using github.com/mvdan/sh to parse the runs: fields and dropping any non-shebang comments.

[jonjohnsonjr]

After much deliberation I think I am going to serialize this as yaml so that it's similar to the input format and more familiar to anyone viewing it. I know there are tools to convert json to yaml, but I feel like this will be a point of friction regardless. The biggest thing for me is that yaml supports multi-line strings, which we use extensively, and having to pipe everything through a conversion tool is annoying.

[smoser]

After much deliberation I think I am going to serialize this as yaml so that it's similar to the input format and more familiar to anyone viewing it. I know there are tools to convert json to yaml, but I feel like this will be a point of friction regardless. The biggest thing for me is that yaml supports multi-line strings, which we use extensively, and having to pipe everything through a conversion tool is annoying.

huh. yaml is a strict superset of json, isn't it?
I tried unsuccessfully to do a more complex example, but this built successfully for me:

python3 -c 'import json, sys, yaml; print(json.dumps(yaml.safe_load(open(sys.argv[1]))) + "\n")' zip.yaml > zip.json
cp zip.json zip.yaml
make package/zip

the zip.json is'nt pretty , but it should be identical when loaded.

{"package": {"name": "zip", "version": 3.0, "epoch": 6, "description": "Creates PKZIP-compatible .zip files", "copyright": [{"license": "Info-ZIP"}]}, "environment": {"contents": {"packages": ["build-base", "busybox", "ca-certificates-bundle", "openssf-compiler-options"]}}, "pipeline": [{"uses": "fetch", "with": {"expected-sha256": "f0e8bb1f9b7eb0b01285495a2699df3a4b766784c1765a8f1aeedf63c0806369", "uri": "https://sourceforge.net/projects/infozip/files/Zip%203.x%20%28latest%29/${{package.version}}/zip30.tar.gz"}}, {"uses": "patch", "with": {"patches": "06-stack-markings-to-avoid-executable-stack.patch 07-fclose-in-file-not-fclose-x.patch 08-hardening-build-fix-1.patch 09-hardening-build-fix-2.patch 10-remove-build-date.patch 12-fix-build-with-gcc-14.patch"}}, {"runs": "make -f unix/Makefile LOCAL_ZIP=\"${CFLAGS} ${CPPFLAGS}\" prefix=/usr generic"}, {"runs": "make -f unix/Makefile prefix=${{targets.destdir}}/usr MANDIR=${{targets.destdir}}/usr/share/man install"}, {"uses": "strip"}], "subpackages": [{"name": "zip-doc", "pipeline": [{"uses": "split/manpages"}], "description": "zip manpages"}], "update": {"enabled": true, "release-monitor": {"identifier": 10080}}, "test": {"pipeline": [{"runs": "zip --version\nzipcloak --version\nzipnote -v\nzipsplit -v\nzip --help\nzipcloak --help\nzipnote -h\nzipsplit -h\n"}]}}

[smoser]

I retact my statement above. you were interested in the readability of the content. which i think is valid.

[mattmoor]

Isn't that @dlorenc password?