[Tracking] TPM 2.0/Secure Boot workstream

[jepio]

Rough first draft, currently not ordered:

• update grub sys-boot/grub: update to 2.06 scripts#1082
• update ignition Upgrade Ignition to version v2.x / spec v3.x, Afterburn to latest upstream #387
• update shim
• update EDK2 OVMF (?)
• investigate SBAT
• run some secure boot enabled tests in CI:
  • support testing secureboot in Qemu mantle#556
• support secure boot testing in mantle #1211
• import tpm2-tools - done together with clevis
• package clevis
  • Add support for TPM- and Tang-based disk encryption scripts#1560
• kernel lockdown:
  • secure boot: lockdown, grub fallback, CI scripts#2299
• include shim in aarch64 build
• ensure clevis works (in ignition or from OS)
• setup signing infrastructure
  • Add the infrastructure for shim signing & aarch64 support scripts#2292
• update pcr policy build infrastructure
• Decide on which Grub to move forward with rhboot/grub2 or vanilla grub (ref [Tracking] TPM 2.0/Secure Boot workstream #630 (comment))
• submit signed shim for review
  • https://github.com/flatcar/shim-review/tree/sayan/add-dockerfile

[jepio]

Mantle now publishes GCP images with UEFI features including vTPM enabled. With Azure CommunityGalleries we can also enable vTPM support on our images. AWS has GA'd their vTPM, but requires switching the AMI to UEFI boot mode (dropping compat with Xen HVM instances). We will need to consider publishing separate ami's for UEFI and BIOS boot.

Secure boot has not been tackled yet, but started thinking about signing procedures and key handling.

[tormath1]

As a side note, Clevis (and friends) are now available on ::guru: https://github.com/gentoo/guru/tree/master/app-crypt/clevis. Including Clevis should then follow the same process as including a package from ::gentoo overlay (https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#add-or-update-a-package).

Butane clevis restriction should also be dropped once done: https://github.com/coreos/butane/blob/87a7a686aff511dd13a6b873baf9a7d005170737/config/flatcar/v1_0/translate.go#L37-L41

[krishjainx]

@tormath1 Trying to add Clevis takes me into dependency hell. I'm essentially doing https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/#add-or-update-a-package but with gentoo/guru instead of gentoo/gentoo

[tormath1]

@krishjainx yes, there are some dependencies like jose and luksmeta that needs to be pulled too. Back in the days @jepio started a PoC for Clevis: https://github.com/flatcar/coreos-overlay/commits/jepio/wip-clevis you can base your work on this (or rebase his branch)

[krishjainx]

The preliminary support for clevis can be found at flatcar/scripts#909. I will return to working on this shortly. I encountered some issues after working on it with an alpha tag and then transitioning it to another branch based on master. So, I will need to rebase and test it with the new SDK container locally. If someone is interested in taking over my work, please feel free to have a look. @tormath1

[jepio]

On EFI we are now missing tpm eventlog access from linux because that depends on the linuxefi boot protocol which is a downstream RH grub patch. I think we'll need to switch to using https://github.com/rhboot/grub2

[jepio]

Gael from matrix shared this link to grub patches for booting an UKI on BIOS systems, which could be interesting: https://github.com/osteffenrh/grub2-blscfg/commits/blscfg-unified-kernel-f38

[chewi]

I've looked into the nature of Red Hat's fork. It's based around Fedora releases with 40 still using 2.06 and 41 moving to 2.12. The patch set is very heavy with 368 commits currently applied on top of 2.12. For some reason, this doesn't quite align with the order or number of patches in their RPM package. There's a lot of noise here, as it includes commits that do things like rename grub to grub2. However, if we need the Linux EFI support, trying to cherry-pick individual commits doesn't seem like a good idea as it's a big change, and it's hard to tell exactly which commits are applicable.

[chewi]

I now realise why I confused about Fedora 41 during our recent meeting. The RPM package has patches against 2.12, but the branch in the rhboot/grub2 repo is still based on 2.06. 😵

We did talk about taking a tarball from GitHub, but I'd also like to use Gentoo's ebuilds if we can because I think little to no changes are needed. That would mean applying Red Hat's changes using a patch, but even as a single file, that patch would be nearly 2MB. 😅 That's not necessarily an issue, but…

[jepio]

Wouldn't a single patch be a nightmare to maintain and rebase? The other thing that is important: we need to stay in control of the sbat.csv as we are responsible for the security fixes to grub.

[chewi]

I did ask about the weird repo situation in Fedora's Matrix, but they say the repos are for different purposes. Things are still in motion though, so maybe this will change soon. Until then, a single patch is the best option. It's actually not that hard, I'll write up the steps.

I've got sbat.csv in hand.

I have now successfully built Red Hat's fork using Gentoo's ebuild. The Gentoo patches conflict, but none of them are applicable to Flatcar, so I've simply unset PATCHES. I've also rebased our own two patches, which took a fair amount of conflict wrangling. The GPT changes are currently failing to build. More soon.

[chewi]

qemu_update and arm64's qemu_uefi passed, but the rest didn't. I tried the image locally, and it does seem like the verity hash isn't getting passed through, at least on amd64. No idea why yet, that patch is quite straightforward.

[chewi]

Still having trouble with this, despite trying a few things. I can rebuild GRUB and write the new grubx64.efi to an existing image, so the turnaround time isn't too bad, but it's still tricky. What's strange is that BIOS is also failing in the same way even though both amd64 and arm64 EFI now use grub-core/loader/efi/linux.c instead. Seemingly it worked on arm64? The result is that systemd waits forever for the usr device, but I can see from using rd.break=pre-mount that the verity argument is missing.

[jepio]

Still having trouble with this, despite trying a few things. I can rebuild GRUB and write the new grubx64.efi to an existing image, so the turnaround time isn't too bad, but it's still tricky. What's strange is that BIOS is also failing in the same way even though both amd64 and arm64 EFI now use grub-core/loader/efi/linux.c instead. Seemingly it worked on arm64? The result is that systemd waits forever for the usr device, but I can see from using rd.break=pre-mount that the verity argument is missing

Can you share your latest code/branch? amd64 and arm64 use different offsets where the verity hash is stored in the vmlinuz file (64 vs 512).
How about we get it working with 2.06 + patches first (since that is closer to what we currently have), and switch to 2.12 in a second step.

[jepio]

Updated the tracker with currently open PRs that are ready to review & merge:

• support testing secureboot in Qemu mantle#556
• secure boot: lockdown, grub fallback, CI scripts#2299

Ready for a early review:

• Add the infrastructure for shim signing & aarch64 support scripts#2292

[chewi]

See flatcar/scripts#2301. The offsets are still respected. I might try 2.06, but I'll try adding some debugging to 2.12 first.

[chewi]

I learnt how to debug GRUB with gdb today. I've fixed this for BIOS, the code needed some adjustment and movement from grub-core/loader/i386/linux.c to grub-core/loader/i386/pc/linux.c. The EFI issue seems to be different.

[jepio]

Now that you mention it: the original verity hash commit did touch grub-core/loader/i386/efi/linux.c but we dropped this when we dropped the patch that introduced linuxefi. This is the first commit (without subsequent fixups): flatcar-hub/grub@03b547c. So this will need to be brought back.

grub-core/loader/i386/pc/linux.c seems to only be relevant to the linux16 command?

[chewi]

Yeah, I was about to say I think this code now needs to go into grub-core/loader/i386/efi/linux.c, which I only noticed a short time ago. My first attempt at this failed, will have another go using your link tomorrow.

[chewi]

I think I've got it now. Time for another Jenkins run.

[chewi]

grub-core/loader/i386/pc/linux.c seems to only be relevant to the linux16 command?

I think linux effectively calls linux16 when the EFI module cannot be loaded.

  cmd_linux =
    grub_register_command ("linux", grub_cmd_linux,
                           0, N_("Load Linux."));
  cmd_linux16 =
    grub_register_command ("linux16", grub_cmd_linux,
                           0, N_("Load Linux."));

[chewi]

I can't see any mention of the TPM event log in Red Hat's patches, only in the vanilla 2.12 sources. Are you sure the patches were needed? 😅

[jepio]

grub-core/loader/i386/pc/linux.c seems to only be relevant to the linux16 command?

I think linux effectively calls linux16 when the EFI module cannot be loaded.

  cmd_linux =
    grub_register_command ("linux", grub_cmd_linux,
                           0, N_("Load Linux."));
  cmd_linux16 =
    grub_register_command ("linux16", grub_cmd_linux,
                           0, N_("Load Linux."));

I think I see what's going on. Mainline grub 2.12 has this in grub-core/Makefile.core.def:

module = {
  name = linux;
  x86 = loader/i386/linux.c;
  (...)
  x86_64_efi = loader/efi/linux.c;

and the fedora-41 branch of rhboot/grub has:

module = {
  name = linux;
  i386_pc = loader/i386/pc/linux.c;
  x86_64_efi = loader/i386/efi/linux.c;
  i386_xen_pvh = loader/i386/linux.c;

That would explain why the patch needs to be adapted for rhboot/grub: vanilla grub compiles in loader/i386/linux.c on both x86 BIOS and EFI, while rhboot grub completely separates them. It would also seem that on rhboot/grub the commands linuxefi and linux are the exact same, so we don't need to change grub.cfg to use a different command.

I can't see any mention of the TPM event log in Red Hat's patches, only in the vanilla 2.12 sources. Are you sure the patches were needed? 😅

It's not so much the patches, as it is the way grub enters the kernel (boot protocol), and this differs between the two branches. I haven't investigated this in detail, but there may be a way to get vanilla grub to enter the x86 kernel on EFI such that the eventlog is exposed. Arm64 efi is simpler and this works in vanilla grub. Compare the lines from arm64 1 and amd64 2:

arm64:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1]
[    0.000000] Linux version 6.6.50-flatcar (build@pony-truck.infra.kinvolk.io) (aarch64-cros-linux-gnu-gcc (Gentoo Hardened 13.3.1_p20240614 p17) 13.3.1 20240614, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP PREEMPT Wed Sep 11 12:18:14 -00 2024
[    0.000000] KASLR enabled
[    0.000000] efi: EFI v2.7 by EDK II
[    0.000000] efi: SMBIOS 3.0=0xdced0000 TPMFinalLog=0xd9760000 MEMATTR=0xdba62198 ACPI 2.0=0xd9700018 TPMEventLog=0xd9673018 RNG=0xd970e698 MEMRESERVE=0xd9b43e18 

amd64:

[    0.000000] Linux version 6.6.50-flatcar (build@pony-truck.infra.kinvolk.io) (x86_64-cros-linux-gnu-gcc (Gentoo Hardened 13.3.1_p20240614 p17) 13.3.1 20240614, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP PREEMPT_DYNAMIC Wed Sep 11 12:15:49 -00 2024
[    0.000000] Command line: BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200 flatcar.first_boot=detected verity.usrhash=45a627c8a86cad518157501cec1c7096df60ba0f4835732718f05affdeecf110
[    0.000000] BIOS-provided physical RAM map:
(...)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] efi: EFI v2.7 by EDK II
[    0.000000] efi: TPMFinalLog=0x9cbf7000 SMBIOS=0x9c9ab000 ACPI=0x9cb7e000 ACPI 2.0=0x9cb7e014 MEMATTR=0x9b678298 

[jepio]

I think I see what's going on. Mainline grub 2.12 has this in grub-core/Makefile.core.def:

Ha, fun stuff: grub-2.06 doesn't have grub-core/loader/efi/linux.c, while vanilla 2.12 does. So it may be that the redhat patch is unnecessary (for what we want to achieve) with vanilla grub-2.12. Would you try it? If you add set debug="linux" to your OEM grub.cfg file then grub prints debug messages for the loader code paths.

[chewi]

Can I tell more directly by looking at the kernel output? Does efi: TPMEventLog= mean that it worked? If so, it's working on my laptop with no effort at all. I guess systemd-boot already supports this!

[chewi]

I've compared my own local amd64 QEMU kernel log between our last release and my recent changes. TPMEventLog appears in the latter but not the former. I'll try without the patches.

[jepio]

Can I tell more directly by looking at the kernel output? Does efi: TPMEventLog= mean that it worked? If so, it's working on my laptop with no effort at all. I guess systemd-boot already supports this!

Yes that's the line we're after. The full test case we need to add to kola is:
tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements successfully reads and parses the event log.
In phase 2 we'll need to make sure that the PCRs match what we expect and can precompute the ones that the distro is responsible for.

[chewi]

I can confirm that efi: TPMEventLog= appears with vanilla(ish) 2.12 and also that sudo tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements works! I guess we'll proceed on that track if you're sure we don't need the Red Hat patches for anything else.

[jepio]

I can't think of anything else we need right now. There may be differences in how grub measures things into PCRs, but we can address that later.

[chewi]

New Kola test is at flatcar/mantle#558.