Introduce new disk partitioning scheme

[unbel13ver]

New disk configuration provides grounds for upcoming features, such as AB software updates and Storage VM and many more.

The Lenovo X1 config has two LVM pools, first one is fixed-size 250G "system" partition (which is going to be encrypted) and the rest of the disk is dedicated to the StorageVM.

Description of changes

Checklist for things done

• Summary of the proposed changes in the PR description
• More detailed description in the commit message(s)
• Commits are squashed into relevant entities - avoid a lot of minimal dev time commits in the PR
• Contribution guidelines followed
• Ghaf documentation updated with the commit - https://tiiuae.github.io/ghaf/
• PR linked to architecture documentation and requirement(s) (ticket id)
• Test procedure described (or includes tests). Select one or more:
  • Tested on Lenovo X1 x86_64
  • Tested on Jetson Orin NX or AGX aarch64
  • Tested on Polarfire riscv64
• Author has run nix flake check --accept-flake-config and it passes
• All automatic Github Action checks pass - see actions
• Author has added reviewers and removed PR draft status

Testing

Current upstream partition scheme is as follows:

[ghaf@ghaf-host:~]$ lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda             8:0    0 465.8G  0 disk 
├─sda1          8:1    0   236M  0 part 
└─sda2          8:2    0   8.6G  0 part 
nvme0n1       259:0    0 953.9G  0 disk 
├─nvme0n1p1   259:1    0     1M  0 part 
├─nvme0n1p2   259:2    0   500M  0 part /boot
└─nvme0n1p3   259:3    0 953.4G  0 part 
  └─pool-root 254:0    0 953.4G  0 lvm  /nix/store
                                        /

To test new partitioning scheme, run lenovo-x1-carbon-gen11-debug-installer image and install Ghaf into the internal SSD storage of Lenovo-X1.
After insallation is completed and the laptop is booted into the Ghaf system, check partitions with lsblk command.
It should be as follows:

[ghaf@ghaf-host:~]$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0 953.9G  0 disk 
├─nvme0n1p1 259:1    0     1M  0 part 
├─nvme0n1p2 259:2    0   500M  0 part /boot
├─nvme0n1p3 259:3    0   500M  0 part 
└─nvme0n1p4 259:4    0    14G  0 part 

Check zpool is online:

[ghaf@ghaf-host:~]$ zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot_1    14G  5.00G  9.00G        -         -     0%    35%  1.00x    ONLINE  -

Check datasets are mounted successfully:

[ghaf@ghaf-host:~]$ mount | grep zroot_1
zroot_1/root_a on / type zfs (rw,relatime,xattr,posixacl,casesensitive,x-initrd.mount)
zroot_1/root_a on /nix/store type zfs (ro,relatime,xattr,posixacl,casesensitive)
zroot_1/storagevm on /storagevm type zfs (rw,relatime,xattr,posixacl,casesensitive)
zroot_1/vm_storage_a on /vm_storage type zfs (rw,relatime,xattr,posixacl,casesensitive)
zroot_1/gp_storage on /gp_storage type zfs (rw,relatime,xattr,posixacl,casesensitive)

Optionally - check there are no failed systemd units:

[ghaf@ghaf-host:~]$ systemctl list-units --all --state=failed
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.

[vilvo]

Please see the detailed comments.

[leivos-unikie]

@unbel13ver can you provide test instructions for this PR? What is the expected outcome of new partitioning scheme?

[unbel13ver]

@unbel13ver can you provide test instructions for this PR? What is the expected outcome of new partitioning scheme?

Testing instructions added.

[leivos-unikie]

• lsblk gives expected output
• ci-test-automation passed
• performance ok
• apps launch

[clayhill66]

With the latest commit 9d7ab74:
-automated cases are ok
-all apps launch - ok
-nixos rebuild does NOT work

[johannarautanen]

[vunnyso]

Reason for limited size is probably because of disko-basic-postboot commands didn't work as expected due to change in partitioning scheme.

[leivos-unikie]

Any idea if this PR can address this problem?
Current mainline ghaf boots partially from USB SSD and nvme if ghaf is installed (with ghaf-installer) to nvme and trying to boot ghaf from USB SSD.

[unbel13ver]

With the latest commit 9d7ab74: -automated cases are ok -all apps launch - ok -nixos rebuild does NOT work

nixos-rebuild does not work for updating partition scheme, in this case the system needs to be re-installed.

Reason for limited size is probably because of disko-basic-postboot commands didn't work as expected due to change in partitioning scheme.

Yes, thanks for noticing this. The disco-basic-postboot.nix does not consider ZFS as a possible filesystem. I need to take a look into that.

[unbel13ver]

Any idea if this PR can address this problem? Current mainline ghaf boots partially from USB SSD and nvme if ghaf is installed (with ghaf-installer) to nvme and trying to boot ghaf from USB SSD.

I am not sure the issue is related to this PR. NixOS picks the device to boot by its label, and in this case there are two devices with the same label (nixos). We got the same issue with Orin devices previously. The solution either to remove external drive or change the label of the internal partition.

[leivos-unikie]

Any idea if this PR can address this problem? Current mainline ghaf boots partially from USB SSD and nvme if ghaf is installed (with ghaf-installer) to nvme and trying to boot ghaf from USB SSD.

I am not sure the issue is related to this PR. NixOS picks the device to boot by its label, and in this case there are two devices with the same label (nixos). We got the same issue with Orin devices previously. The solution either to remove external drive or change the label of the internal partition.

Actually nowadays 'LABEL' is empty for every partition, they have 'PARTLABEL' though.

I made a test, changed the partlabels of partitions on nvme with gdisk (nix-shell -p gptfdisk). Then tried to boot ghaf from USB SSD. Still it booted partially from USB and nvme.

So it seems that the boot is not by LABEL nor by PARTLABEL.

After installing ghaf to nvme the only way for booting ghaf from USB SSD is to boot first non-ghaf OS from USB SSD and wipe nvme.

Is it ok to discuss this here, or should we move elsewhere...

[leivos-unikie]

So it seems that the boot is not by LABEL nor by PARTLABEL.

Well, not quite so. If there is no disk-disk1-ESP partlabel available on any device the boot fails with timeout (not finding /dev/disk/by-partlabel/disk-disk1-ESP). Other partlabels seem to be insignificant, for example:

[unbel13ver]

ZFS partition is now extended during the first boot.
This example is when the system is booted from the external SSD (root is on the /dev/sda):

[ghaf@ghaf-host:~]$ lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda       8:0    0 465.8G  0 disk 
├─sda1    8:1    0     1M  0 part 
├─sda2    8:2    0   500M  0 part /boot
├─sda3    8:3    0   500M  0 part 
└─sda4    8:4    0 464.8G  0 part 
nvme0n1 259:0    0 953.9G  0 disk 

[ghaf@ghaf-host:~]$ zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfspool   464G  5.04G   459G        -         -     0%     1%  1.00x    ONLINE  -

I have also renamed zfs pool from zroot_1 to zfspool since there is only one zfs pool in use.

[milva-unikie]

Tested on Lenovo-X1

Everything seems to be good with both installer and debug-image! (I did not test nixos-rebuild though, please let me know if that needs to be still checked)

• Installer installed to internal SSD

  • Installation works
  • Test-automation passes
  • The iso-image is a bit bigger than it was previously (8,5 GB now vs 7,3 GB previously)
  • Same testing results as in Testing-section except that zroot_1 has been renamed to zfspool

• Debug-image installed to external SSD

  • Test-automation passes
  • disk1.raw.zst is a bit bigger than it was previously (4,5 GB now vs 3,8 GB previously), but the extracted image is the same size (16,1 GB)

(As a note for our testing team: The free space in host is reduced with this PR and there is no longer an excessive amount of free space to run fileio-tests. We need to decide how we want to proceed with that.)