github/workflows: Sign Ubuntu and Arch images using cosign

[travier]

Based on / waiting for #1439

Needs a key to be generated and imported as secret.

[softwarefactory-project-zuul]

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/64eff396857943779e506c0443e9cf1b

❌ unit-test FAILURE in 6m 49s
❌ unit-test-migration-path-for-coreos-toolbox FAILURE in 3m 10s
❌ unit-test-restricted FAILURE in 5m 09s
✔️ system-test-fedora-rawhide SUCCESS in 29m 58s
✔️ system-test-fedora-39 SUCCESS in 28m 50s
✔️ system-test-fedora-38 SUCCESS in 30m 30s

[Foxboron]

Would toolbox validate the signatures with the current library use, or is there more code needed for that?

[softwarefactory-project-zuul]

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/69e6cc97da26471c96665fb4e8553b52

✔️ unit-test SUCCESS in 5m 41s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 19s
✔️ unit-test-restricted SUCCESS in 5m 39s
✔️ system-test-fedora-rawhide SUCCESS in 31m 18s
✔️ system-test-fedora-39 SUCCESS in 26m 58s
✔️ system-test-fedora-38 SUCCESS in 29m 28s

[travier]

Would toolbox validate the signatures with the current library use, or is there more code needed for that?

This currently only works with podman with the following setup: https://github.com/toolbx-images/images#verifying-sigstore-container-signatures-with-podman

[Foxboron]

Would toolbox validate the signatures with the current library use, or is there more code needed for that?

This currently only works with podman with the following setup: https://github.com/toolbx-images/images#verifying-sigstore-container-signatures-with-podman

This looks like something that toolbox should support natively when pulling images. Should probably look into that.

[travier]

Would toolbox validate the signatures with the current library use, or is there more code needed for that?

This currently only works with podman with the following setup: toolbx-images/images#verifying-sigstore-container-signatures-with-podman

This looks like something that toolbox should support natively when pulling images. Should probably look into that.

Toolbox calls out to podman to pull images. It does not do it by itself. Thus why setting it up for podman makes it work.

One option would be to ask for the podman maintainers to extend this format to be able to include drop-ins for example in the policy to let packages add their own keys & policy for example. But that might be difficult to do safely.

[Foxboron]

Toolbox calls out to podman to pull images. It does not do it by itself. Thus why setting it up for podman makes it work.

Ah, I see It actually shells out to podman instead of using the underlying Go libraries. Obviously a bunch would have to be rewritten but it would offer more flexibility.

[softwarefactory-project-zuul]

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/2fd65595647d469ea6e7572acb3bea31

✔️ unit-test SUCCESS in 6m 35s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 18s
✔️ unit-test-restricted SUCCESS in 5m 51s
✔️ system-test-fedora-rawhide SUCCESS in 33m 27s
✔️ system-test-fedora-39 SUCCESS in 30m 33s
✔️ system-test-fedora-38 SUCCESS in 31m 03s

[debarshiray]

Based on / waiting for #1439

Needs a key to be generated and imported as secret.

Thanks for working on this, @travier ! Do you need something to generate the key and import it as a secret? You should have admin access to this repository.

[debarshiray]

Would toolbox validate the signatures with the current library use, or is there more code needed for that?

This currently only works with podman with the following setup: toolbx-images/images#verifying-sigstore-container-signatures-with-podman

I see that the JSON is documented in containers-policy.json(5), which is great.

This looks like something that toolbox should support natively when pulling images. Should probably look into that.

Toolbox calls out to podman to pull images. It does not do it by itself. Thus why setting it up for podman makes it work.

One option would be to ask for the podman maintainers to extend this format to be able to include drop-ins for example in the policy to let packages add their own keys & policy for example. But that might be difficult to do safely.

I see that on Fedora, the containers-common package already ships a /etc/containers/policy.json that includes registry.access.redhat.com. Does it mean that we are already verifying the authenticity of our UBI-based RHEL images?

I wonder if we can add the quay.io/toolbx images to that. What about the registry.fedoraproject.org images? Are they signed? If not, maybe we should look into that?

It looks like podman pull doesn't allow specifying a specific JSON file, even though containers-policy.json(5) suggests that tools might offer that. Would it help if toolbox(1) could supply it's own JSON to podman pull? We could at least verify the images for which we offer built-in support.

[travier]

Let's merge #1439 first before we look at this one?

[travier]

Thanks for working on this, @travier ! Do you need something to generate the key and import it as a secret? You should have admin access to this repository.

This is the whole question: Who should generate and store a backup of the key? Should this be a Fedora key? I don't think there are cosign keys in Fedora infra or support for cosign there yet.

[travier]

Does it mean that we are already verifying the authenticity of our UBI-based RHEL images?

I think yes, we are, but using GPG keys.

What about the registry.fedoraproject.org images? Are they signed? If not, maybe we should look into that?

Not sure but I don't think they are signed. They don't show up as signed in https://quay.io/repository/fedora/fedora?tab=tags

It looks like podman pull doesn't allow specifying a specific JSON file, even though containers-policy.json(5) suggests that tools might offer that. Would it help if toolbox(1) could supply it's own JSON to podman pull? We could at least verify the images for which we offer built-in support.

That coud be another option but that would also likely need discussion with the podman folks.

[debarshiray]

Let's merge #1439 first before we look at this one?

Yes, definitely. I was away for the past week, and I threw some questions that came to me while I was gone.

[debarshiray]

Thanks for working on this, @travier ! Do you need something to generate the key and import it as a secret? You should have admin access to this repository.

This is the whole question: Who should generate and store a backup of the key? Should this be a Fedora key? I don't think there are cosign keys in Fedora infra or support for cosign there yet.

If we consider only the images for Arch Linux and Ubuntu, then maybe we can do the same thing as you were doing before for them at github.com/toolbx-images/images? Clearly, I don't know much about cosign, so I will just follow your lead.

I only brought up the Fedora and RHEL images to understand the broader status quo, but we don't have to solve them as part of this pull request.

Does it mean that we are already verifying the authenticity of our UBI-based RHEL images?

I think yes, we are, but using GPG keys.

I see. I think I need to read up to understand what cosign versus GPG means, and how they compare, etc.. :)

[Foxboron]

I see. I think I need to read up to understand what cosign versus GPG means, and how they compare, etc.. :)

TL;DR:

cosign (read this as sigstore) intends to create ephemeral short-lived keys through workload identities or Open-ID Connect identities. They are tied to the fulcio CA and all signatures are appended to a Transparency Log which will allows you detect signing key misuse. You can also sign with your own keys if you want.

In comparison GnuPG is just a signature.