Contrib: Deprecate torcx, ship containerd / docker as sysexts

[t-lo]

This PR supersedes #982 which has been idling for a while. Main motivation of re-issuing the PR is to provide a branch directly on the scripts repo for Flatcar maintainers to collaborate on.

Features added

The PR includes a number of improvements to streamline the developer experience with sysexts. The improvements are:

• build_sysext can now handle dependencies and create sysexts that depend on packages of other sysexts
• build_image takes a command line option of the sysexts to include in the base OS, defaulting to containerd and docker. Motivation is to make it easier down the road to move base OS packages into sysexts, and to build leaner OS images.

Testing

• Github actions CI / test: https://github.com/flatcar/scripts/actions/runs/6614425214/job/17964395526?pr=1216
• Jenkins CI: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/2791/cldsv/

Unrelated minor improvements

• The PR also includes a few unrelated QoL improvements which were implemented along the way to ease development work and testing of this PR:
  • Add run_local_tests.sh to lower the bar of running tests locally for the latest image built
  • Make the update payload configurable for the qemu_update.sh test case. Formerly it was always fetched from bincache; fetch failures were silently ignored and lead to this test being skipped.
  • Add a new flag to run_sdk_container to not touch / update the version file (-U)
  • Fix detection of "SDK container image exists in local Docker registry and is up to date" in ci-automation. Formerly, this check always failed for nightly builds, leading to the SDK container being re-downloaded each time an SDK container was newly instanciated (e.g. after running docker container prune).

Dependencies (squashfs zstd support for Jenkins CI nodes)

• add zstd support to squashfs (release 3602) #1297
• add zstd support to squashfs (release 3732) #1296
• add zstd support to squashfs (release 3745) #1295
• https://github.com/flatcar/flatcar-linux-infra/pull/268

[ALL DONE] Todo items

• Update and rename torcx mantle / kola tests to handle sysext, add test for disabling docker: Kola: update to handle torcx deprecation / docker and containerd sysext mantle#465
• Remove torcx logic and command line options from build_packages, build_image, and friends
• Add SLSA provenance generation to docker / containerd sysext builds and ensure SPDX SBOMs, file lists, and licenses are covered
• Update github actions CI and kola workflows to not handle torcx anymore
• Update docker docs https://www.flatcar.org/docs/latest/container-runtimes/getting-started-with-docker/#permanently-running-a-container to include a soft-link /etc/systemd/system/multi-user.target.wants/docker.service => /usr/lib/systemd/system/docker.service for sysext based docker. See kola/tests/docker/enable_docker.go for details. : Torcx removal: update documentation flatcar-archive/flatcar-docs#343
• Remove torcx sanity checks in update_engine post-install script: flatcar-postinst: Remove torcx sanity checks update_engine#30

[jepio]

@pothos does this work?

[t-lo]

I also don't quite understand why we first create a docker sysext pretending containerd is installed via ${PORTAGE_CONFIGROOT}/etc/portage/profile/package.provided and then create a containerd sysext anyway.

[pothos]

That was done to have a split containerd and Docker sysext setup. This way users can disable Docker but keep our containerd.

[pothos]

(The pretending of containerd being installed is to build the Docker sysext without containerd in it.)

[jepio]

I mean whether the symlinking to etc at this phase of the build will work

[t-lo]

@jepio to answer your question this works because of

scripts/build_library/build_image_util.sh

Lines 793 to 803 in 7c25545

	# Backup the /etc contents to /usr/share/flatcar/etc to serve as
	# source for creating missing files. Make sure that the preexisting
	# /usr/share/flatcar/etc does not have any meaningful (non-empty)
	# files, so we remove nothing important. There shouldn't be any
	# symlinks either. Add "! -type d" to exclude directories as "stat"
	# usually returns a size of a directory being 4096 or so.
	if [[ $(sudo find "${root_fs_dir}/usr/share/flatcar/etc" -size +0 ! -type d 2>/dev/null | wc -l) -gt 0 ]]; then
	die "Unexpected non-empty files in ${root_fs_dir}/usr/share/flatcar/etc"
	fi
	sudo rm -rf "${root_fs_dir}/usr/share/flatcar/etc"
	sudo cp -a "${root_fs_dir}/etc" "${root_fs_dir}/usr/share/flatcar/etc"

.

[t-lo]

That was done to have a split containerd and Docker sysext setup. This way users can disable Docker but keep our containerd.

Oh I understand the intent, but I'd argue that if we want stackable / dependent sysexts then we should make the packages shipped with each layer explicit, and re-use these when we build the next "upper" layer.

I've extended build_sysext.sh to cover this, and updated build_image_util.sh to use it.

[jepio]

That looks more complex than it needs to be. How about just mount the whole previous sysext when building the next one.

[t-lo]

Sysexts do not contain package information since they only cover /usr (and/or /opt). Installed packages are registered in sub-directories below /var/db/pkg.
(I wish this was easier though!)

[jepio]

this looks unused

[t-lo]

Yeah, the whole thing needs some cleaning up. This was added in a no-comment commit without justification, and the environment variable does not appear anywhere else in the code.

[t-lo]

The PR in its current state:

• can build OS images
• does not build any torcx packages or artifacts
• includes docker and containerd sysexts in OS image and update tarball, which have been manually tested and are working (qemu)

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/6688993401

[t-lo]

Added SBOM, licenses, packages and file listing / disk usage inventory information for sysexts.

• SLSA provenance is included with the sysexts.
• SBOM and licenses JSON now include all packages of the final image, i.e. a combined list of base image and all base OS sysexts.
• Packages lists, files list and detailed files list separate sections with files /packages lists for each sysext. The files and detailed files lists include tha sysext squashfs files on the base image.
• Disk usage contains both final disk image usage as well as usage of each individual sysext squashfs.

Sample files attached - @krnowak @jepio @pothos could you please have a look and tell me if that works for you?

flatcar_production_image_contents.txt
flatcar_production_image_contents_wtd.txt
flatcar_production_image_disk_usage.txt
flatcar_production_image_licenses.json
flatcar_production_image_packages.txt
flatcar_production_image_sbom.json

[pothos]

Please change the name app-containers_docker.raw and app-containers_containerd.raw to docker-flatcar.raw and containerd-flatcar.raw. We have documented those for ages in
https://www.flatcar.org/docs/latest/container-runtimes/use-a-custom-docker-or-containerd-version/
and
https://www.flatcar.org/docs/latest/provisioning/sysext/#supplying-your-sysext-image-from-ignition
for a smooth upgrade experience which doesn't break user setups.

Also, app-containers is a gentooism that isn't really meaningful to users. While finally better than app-emulation we just shouldn't use Gentoo packaging categories for user-facing names. Also, we might want to add more things into the mix and have other Flatcar extensions as well, and the use of Gentoo packaging categories is not a good idea for consistency.

[pothos]

flatcar_production_image_contents.txt

The concatenation is not good for automated processing. We use this file for the image change reports and would have to split all these ### Sysext app-containers_docker.raw headers. Let's have separate files, instead of making our life harder.

[t-lo]

flatcar_production_image_contents.txt

The concatenation is not good for automated processing. We use this file for the image change reports and would have to split all these ### Sysext app-containers_docker.raw headers. Let's have separate files, instead of making our life harder.

Listing sysext contents in these files in separate sections was an explicit ask. We additionally ship separate files per sysext, so processing everything should be rather simple: stop parsing flatcar_production_image_contents.txt at the first ^#.

[t-lo]

Please change the name app-containers_docker.raw and app-containers_containerd.raw to docker-flatcar.raw and containerd-flatcar.raw. We have documented those for ages in https://www.flatcar.org/docs/latest/container-runtimes/use-a-custom-docker-or-containerd-version/ and https://www.flatcar.org/docs/latest/provisioning/sysext/#supplying-your-sysext-image-from-ignition for a smooth upgrade experience which doesn't break user setups.

Also, app-containers is a gentooism that isn't really meaningful to users. While finally better than app-emulation we just shouldn't use Gentoo packaging categories for user-facing names. Also, we might want to add more things into the mix and have other Flatcar extensions as well, and the use of Gentoo packaging categories is not a good idea for consistency.

Got it, will update. It's a bit sad because this convention will make it harder to generically split more apps from the base image into sysexts since we need to maintain app -> name relationships for each, while the current implementation guarantees unique names.

[krnowak]

flatcar_production_image_contents.txt

The concatenation is not good for automated processing. We use this file for the image change reports and would have to split all these ### Sysext app-containers_docker.raw headers. Let's have separate files, instead of making our life harder.

Listing sysext contents in these files in separate sections was an explicit ask. We additionally ship separate files per sysext, so processing everything should be rather simple: stop parsing flatcar_production_image_contents.txt at the first ^#.

Or maybe even better - create a separate file, like flatcar_production_image_contents_kitchen_sink.txt, and leave other files as they were.

[t-lo]

flatcar_production_image_contents.txt

The concatenation is not good for automated processing. We use this file for the image change reports and would have to split all these ### Sysext app-containers_docker.raw headers. Let's have separate files, instead of making our life harder.

Listing sysext contents in these files in separate sections was an explicit ask. We additionally ship separate files per sysext, so processing everything should be rather simple: stop parsing flatcar_production_image_contents.txt at the first ^#.

Or maybe even better - create a separate file, like flatcar_production_image_contents_kitchen_sink.txt, and leave other files as they were.

That's what I had initially, then the request came up to create merged lists. I'm not complaining though, the way you propose it is simpler and requires less code. Will update the PR.

[pothos]

Maybe the communication about the files got confusing because they serve different purposes and land on different places. For the license JSON it makes sense to have a merged view while for the contents txt files it rather doesn't because they are developer-focused. For the SBOM I think the merged view makes sense, too (even if it doesn't truly reflect what the user finds when opening the image, but I think that live SSH interaction for a scanner would also be a valid use case, so better list the files than not).

[t-lo]

Updated the PR. Please find new inventory sample files attached.
flatcar_production_image_contents.txt
flatcar_production_image_contents_wtd.txt
flatcar_production_image_disk_usage.txt
flatcar_production_image_licenses.json
flatcar_production_image_packages.txt
flatcar_production_image_sbom.json
containerd-flatcar_contents.txt
containerd-flatcar_contents_wtd.txt
containerd-flatcar_disk_usage.txt
containerd-flatcar_packages.txt
docker-flatcar_contents.txt
docker-flatcar_contents_wtd.txt
docker-flatcar_disk_usage.txt
docker-flatcar_packages.txt

[t-lo]

Most notably I extended the format of --base-sysexts= in build_image to support sysext names independent of the packages included. The parameter currently defaults to containerd-flatcar:app-containers/containerd,docker-flatcar:app-containers/docker so we get a containerd sysext named containerd-flatcar.raw and a docker sysext named docker-flatcar.raw.

[t-lo]

Addressed Krzesimir's comments, rebased to latest main, restarted Ci tests.

[t-lo]

• Github CI is all green
• Jenkins CI is all green
• 2 LGTMs

==> Merging PR. Thank you so much @pothos and @krnowak for your help and support with improving this PR and getting it merged! Also big thanks to @tormath1 for help with the test suite and to @krishjainx for starting this work!

[pothos]

Thanks, somehow I assumed the kernel config change would be part of last week's releases and also that we should have this here in Alpha. Eagerly waiting for the next Alpha then^^