Image Builder vNext

[richardcase]

Background

Whilst adding support for Cloud Hypervisor it has become apparent that our current approach isn't very scalable or customizable.

We are in a situation where kernel modules for each kernel version we support have to be embedded into our ubuntu base image. Whilst we only support Firecracker this was manageable (just) but now we have Cloud Hypervisor support the number of modules we have to embed has increased dramatically.

This also means we are in a situation where there is a dependency between all our images kernel<-ubuntu base<-capmvm. For anyone to bring their own kernel image this means they need to full understand this dependency chain.....and replicate it.

Also, we are maintaining our own Cluster API base image which is separate from the official community Cluster API images.

Recommended Approach

1. Embed the kernel modules into the kernel image (or create a separate kernel modules image) (DONE)
2. Change the ubuntu-base image to remove all kernel modules (DONE)
3. Use the officical CAPI image as the base for our capmvm image. This requires:
   a. Building the raw capi image from thge k8s image-builder.....make build-raw-ubuntu-2004
   b. Use fdisk -l to get the offset of the partition and block size
   c. Use the offset to mount the raw image sudo mount -o offset=2048 ubuntu-2004-kube-v1.20.10 ./mnt
   d. Import the mounted file into a new container: sudo tar -C mnt -c . | docker import - mycapi:v1.20.10
   e. Publish image
4. Update flintlock to support mounting additional volumes based on a mount point - see this commit for an example (DONE)
5. Update capmvm so that it adds the mount details for the kernel modules so they are in the correct place in the guest vm. For example: (DONE)

      volumes:
      - id: modules
        image: "docker.io/richardcase/flintlock-kernel-modules:5.10.77"
        mountPoint: "/lib/modules/5.10.77" 

We could also have a better UX in CAPMVM (or flintlock) where the kernel image name can be defaulted and the modules volume automatically added.

[yitsushi]

So if I understand it, with this approach, a VMSpec should have 3 images defined:

1. Kernel (kernel version x.y.z)
2. KernelModules mounted under /lib/modules/x.yz
3. RootFS

Potentially we can provide:

• docker.io/weaveworks-liquidmetal/flintlock-kernel:5.10.77
• docker.io/weaveworks-liquidmetal/flintlock-kernel:5.15.65
• docker.io/weaveworks-liquidmetal/flintlock-kernel-modules:5.10.77
• docker.io/weaveworks-liquidmetal/flintlock-kernel-modules:5.15.65

But a user can build and use their own kernel or modules (or both) images.

the number of modules we have to embed has increased dramatically.

How does it reduce the number of kernel versions and modules we have to take care of? Or is it mostly for "organizing stuff better"?

[richardcase]

Good questions 😄

It doesn't reduce the number of kernel versions but it separates our kernel images from our base images. So we don't have to embed the modules from all supported kernels into our base ubuntu image.

So I think it's mostly about:

• Better separation / organization
• Easier to bring your own kernel (as you don't have to build your own base / capmvm images as well)
• Using the standard community capi image

[Callisto13]

4. Update flintlock to support mounting additional volumes based on a mount point - see this commit for an example

There is no diff here, could you link where it is?

[Callisto13]

@richardcase the image created from the capi ubuntu raw build is more than 4G big... feels a bit excessive for an ultralight metal environment 😅
out current OS images are just over 1G
we might want to get some version of our own images into the capi image builder repo, no?

[Callisto13]

Have you looked at the boot times as well as the size?

No did not look at that yet, will try to find time today or when I am back after new year

On average mvms with the capi base images are about a second and a half slower

[richardcase]

Thats good to know 👍

[richardcase]

So, should we have both the current image and the capi image? For some people 1.5 seconds isn't a lot but having the same base image across their clusters in different infra may be more important.

[Callisto13]

mhhhmmm not massively keen on having more stuff to maintain, and kind of defies the reason why we wanted to do this (ie, maintain less, keep in line with community). that would bring us up to 7 (plus all the versions):

• kernel builder image
• kernel binary image
• kernel mods image
• (kernel all in one image, keeping for a bit for backwards compat)
• our base image
• the capi base image (not that we have to write any of it, but we will want some "click this button" automation to get a new one published in our packages when there is a new k8s version)
• k8s image from our base image
• k8s image from capi base image (just a couple of things we need to add there since all the bins are installed in the base)

ideally i would rather get a "stripped down" version into the capi image builder because 4gb feels like a bit of a joke. but that is a bit/lot more work

[Callisto13]

Okay I can get the "k8s image from capi base image" (or just our version of the capi base image) down a bit by removing things we don't need after the fact. Of course the machine image has a bunch of kernel stuff which we provide elsewhere, and the apt cache is full of stuff, lots of little things like this.
Another thing that is making the capi image quite large is that it comes with all the container images for the cluster baked in. This may be useful for us... or not. Although the base OS image for mvms will be very large, if they all have k8s images in those base layers, then they won't need to download them individually thus saving space on the machine overall.
I have yet to test that this actually works like that, idk if the containerds running in the mvms will see those pre-baked snapshots and images as usable or not. Answer: no. base layers do not work like that for vms. if you are starting with a 4.2g os, that amount will be claimed from your mvm disk.
Also those images only count for just shy of 1g of space. So there is another 2 or so of cruft i would like to remove.

Buuut I have not yet found what is causing the slow(er) boots. Hoping I might stumble upon it while removing things. Tho having the cluster images already there will probs make a node boot faster. space or speed is the thing... which is a shame. Because the sort of environments we are targeting will require to be good at both.