Support OEM systemd-sysext images and Flatcar extensions #1016

pothos · 2023-07-26T11:22:00Z

The vendor tools on the OEM partition weren't updated. We now want to
ship them as systemd-sysext images which we can easily update. This
change extends the Flatcar A/B update mechanism to cover the OEM
systemd-sysext images. The same mechanism is also able to support
"official" Flatcar extensions, e.g., a ZFS extension.

Additional change:

GitHub Actions: Use correct version in kola run and skip update test
The kola run didn't pick up the version that was set up in the build
because the git changes from that step are lost.
Redo the version setup in the kola run to use the same version, and
skip the kola update test if no update payload can be found. In the
future we should copy it over from the GitHub Action artifact.

How to use

After merging we always need to publish signed OEM payloads during release.

Used PRs:
flatcar/update_engine#24
flatcar/mantle#448
flatcar/bootengine#68
flatcar/init#101

When merging, create a follow-up GitHub issue for proper update tests on GitHub Actions PR builds (only for the regular update test, that should be easily doable). Done: flatcar/Flatcar#1172

Testing done

See update-engine PR, Jenkins run is here

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

sdk_container/src/third_party/coreos-overlay/coreos-base/misc-files/misc-files-0.ebuild

github-actions · 2023-09-06T15:09:43Z

Test report for 3718.0.0+nightly-20230905-2100 / amd64 arm64

Platforms tested : qemu_uefi-amd64 qemu_update-amd64 qemu_uefi-arm64 qemu_update-arm64

bpf.execsnoop 🟢 Succeeded: qemu_uefi-amd64 (1)

bpf.local-gadget 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.basic 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cgroupv1 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cloudinit.basic 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cloudinit.multipart-mime 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cloudinit.script 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid0.data 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid0.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid1.data 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid1.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.etcd-member.discovery 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.etcd-member.etcdctlv3 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.etcd-member.v2-backup-restore 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.filesystem 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.flannel.udp 🟢 Succeeded: qemu_uefi-amd64 (1)

cl.flannel.vxlan 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.instantiated.enable-unit 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.kargs 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.luks 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.indirect 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.indirect.new 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.regular 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.regular.new 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.reuse 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.wipe 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.symlink 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.translation 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.btrfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.ext4root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.groups 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.once 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.sethostname 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.users 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.xfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.btrfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.ext4root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.users 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.xfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2_1.ext4checkexisting 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2_1.swap 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2_1.vfat 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.install.cloudinit 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.internet 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.locksmith.cluster 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.misc.falco 🟢 Succeeded: qemu_uefi-amd64 (1)

cl.network.initramfs.second-boot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.listeners 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.wireguard 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.omaha.ping 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.osreset.ignition-rerun 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.overlay.cleanup 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.swap_activation 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.sysext.boot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.sysext.fallbackdownload # SKIP 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.toolbox.dnf-install 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.update.badverity 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.update.grubnop 🟢 Succeeded: qemu_uefi-amd64 (1)

cl.update.reboot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.users.shells 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.verity 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.auth.verify 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.groups 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.once 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.resource.local 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.resource.remote 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.resource.s3.versioned 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.security.tls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.sethostname 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.systemd.enable-service 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.locksmith.reboot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.locksmith.tls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.selinux.boolean 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.selinux.enforce 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.tls.fetch-urls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.update.badusr 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

devcontainer.docker 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

devcontainer.systemd-nspawn 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.btrfs-storage 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.containerd-restart 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.lib-coreos-dockerd-compat 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.network 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.selinux 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.torcx-manifest-pkgs 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.userns 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.24.14.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (2) ❌ Failed: qemu_uefi-arm64 (1)

Diagnostic output for qemu_uefi-arm64, run 1

    L1: "  "
    L2: " Error: _cluster.go:117: I0907 19:11:36.499328    1488 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.24"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.24.17"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.24.17"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.24.17"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.24.17"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.7"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L9: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.8.6"
    L10: "cluster.go:117: I0907 19:11:50.610103    1652 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.24"
    L11: "cluster.go:117: [init] Using Kubernetes version: v1.24.17"
    L12: "cluster.go:117: [preflight] Running pre-flight checks"
    L13: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L14: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L15: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L16: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L17: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L18: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L19: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.2?5]"
    L20: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L22: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L27: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L28: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L29: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L30: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L35: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L36: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L37: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L40: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L41: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L42: "cluster.go:117: [apiclient] All control plane components are healthy after 8.004084 seconds"
    L43: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L44: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L45: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L47: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]"
    L48: "cluster.go:117: [bootstrap-token] Using token: 4gy6vy.g3lq3ls4dnhygq0s"
    L49: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L53: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L54: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L55: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L56: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L57: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L58: "cluster.go:117: "
    L59: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L60: "cluster.go:117: "
    L61: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L62: "cluster.go:117: "
    L63: "cluster.go:117:   mkdir -p $HOME/.kube"
    L64: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L65: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L66: "cluster.go:117: "
    L67: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L68: "cluster.go:117: "
    L69: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L70: "cluster.go:117: "
    L71: "cluster.go:117: You should now deploy a pod network to the cluster."
    L72: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L73: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L74: "cluster.go:117: "
    L75: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L76: "cluster.go:117: "
    L77: "cluster.go:117: kubeadm join 10.0.0.25:6443 --token 4gy6vy.g3lq3ls4dnhygq0s _"
    L78: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:94d005552f5afefe73a36af719e1d50586b41c2b74d8730b9779eba078874c53 "
    L79: "cluster.go:117: namespace/tigera-operator created"
    L80: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created"
    L81: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created"
    L82: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created"
    L83: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created"
    L84: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created"
    L85: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created"
    L86: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created"
    L87: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created"
    L88: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created"
    L89: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created"
    L90: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created"
    L91: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created"
    L92: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created"
    L93: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created"
    L94: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created"
    L95: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created"
    L96: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created"
    L97: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created"
    L98: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created"
    L99: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created"
    L100: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created"
    L101: "cluster.go:117: serviceaccount/tigera-operator created"
    L102: "cluster.go:117: clusterrole.rbac.authorization.k8s.io/tigera-operator created"
    L103: "cluster.go:117: clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created"
    L104: "cluster.go:117: deployment.apps/tigera-operator created"
    L105: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io condition met"
    L106: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io condition met"
    L107: "cluster.go:117: installation.operator.tigera.io/default created"
    L108: "cluster.go:117: apiserver.operator.tigera.io/default created"
    L109: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L110: "harness.go:582: Found emergency shell on machine 3342f113-4c12-4874-9939-922291d6cc82 console"
    L111: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine 3342f113-4c12-4874-9939-922291d6cc82 console"
    L112: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39migni???0m - Ignition (record completion). ) on machine 3342f113-4c12-4874-9939-922291d6cc82 console_"
    L113: " "

kubeadm.v1.24.14.calico.cgroupv1.base 🟢 Succeeded: qemu_uefi-amd64 (2); qemu_uefi-arm64 (1) ❌ Failed: qemu_uefi-amd64 (1)

Diagnostic output for qemu_uefi-amd64, run 1

    L1: " Error: _cluster.go:117: I0907 19:13:31.225530    1662 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.24"
    L2: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.24.17"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.24.17"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.24.17"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.24.17"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.7"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.8.6"
    L9: "cluster.go:117: I0907 19:13:42.192932    1830 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.24"
    L10: "cluster.go:117: [init] Using Kubernetes version: v1.24.17"
    L11: "cluster.go:117: [preflight] Running pre-flight checks"
    L12: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L13: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L14: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L15: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L16: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L17: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L18: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.4?2]"
    L19: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L20: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L22: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L27: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L28: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L29: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L30: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L35: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L36: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L37: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L40: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L41: "cluster.go:117: [apiclient] All control plane components are healthy after 5.501893 seconds"
    L42: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L43: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L44: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L45: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]"
    L47: "cluster.go:117: [bootstrap-token] Using token: 3zn0yd.24zbmgclz4eh15so"
    L48: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L49: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L53: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L54: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L55: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L56: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L57: "cluster.go:117: "
    L58: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L59: "cluster.go:117: "
    L60: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L61: "cluster.go:117: "
    L62: "cluster.go:117:   mkdir -p $HOME/.kube"
    L63: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L64: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L65: "cluster.go:117: "
    L66: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L67: "cluster.go:117: "
    L68: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L69: "cluster.go:117: "
    L70: "cluster.go:117: You should now deploy a pod network to the cluster."
    L71: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L72: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L73: "cluster.go:117: "
    L74: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L75: "cluster.go:117: "
    L76: "cluster.go:117: kubeadm join 10.0.0.42:6443 --token 3zn0yd.24zbmgclz4eh15so _"
    L77: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:4531761de364769948f3c55174bf5ddb43ad64dba046119f61a20faabe697aba "
    L78: "cluster.go:117: namespace/tigera-operator created"
    L79: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created"
    L80: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created"
    L81: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created"
    L82: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created"
    L83: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created"
    L84: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created"
    L85: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created"
    L86: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created"
    L87: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created"
    L88: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created"
    L89: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created"
    L90: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created"
    L91: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created"
    L92: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created"
    L93: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created"
    L94: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created"
    L95: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created"
    L96: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created"
    L97: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created"
    L98: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created"
    L99: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created"
    L100: "cluster.go:117: serviceaccount/tigera-operator created"
    L101: "cluster.go:117: clusterrole.rbac.authorization.k8s.io/tigera-operator created"
    L102: "cluster.go:117: clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created"
    L103: "cluster.go:117: deployment.apps/tigera-operator created"
    L104: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io condition met"
    L105: "cluster.go:117: customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io condition met"
    L106: "cluster.go:117: installation.operator.tigera.io/default created"
    L107: "cluster.go:117: apiserver.operator.tigera.io/default created"
    L108: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L109: "--- FAIL: kubeadm.v1.24.14.calico.cgroupv1.base/nginx_deployment (92.54s)"
    L110: "kubeadm.go:313: nginx is not deployed: ready replicas should be equal to 1: null_"
    L111: " "
    L112: "  "

kubeadm.v1.24.14.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.24.14.cilium.cgroupv1.base 🟢 Succeeded: qemu_uefi-amd64 (2); qemu_uefi-arm64 (1) ❌ Failed: qemu_uefi-amd64 (1)

Diagnostic output for qemu_uefi-amd64, run 1

    L1: " Error: _cluster.go:117: I0907 19:21:13.301451    1596 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.24"
    L2: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.24.17"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.24.17"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.24.17"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.24.17"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.7"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.8.6"
    L9: "cluster.go:117: I0907 19:21:23.262034    1758 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.24"
    L10: "cluster.go:117: [init] Using Kubernetes version: v1.24.17"
    L11: "cluster.go:117: [preflight] Running pre-flight checks"
    L12: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L13: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L14: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L15: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L16: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L17: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L18: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.1?39]"
    L19: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L20: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L22: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L27: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L28: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L29: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L30: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L35: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L36: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L37: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L40: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L41: "cluster.go:117: [apiclient] All control plane components are healthy after 5.503058 seconds"
    L42: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L43: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L44: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L45: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]"
    L47: "cluster.go:117: [bootstrap-token] Using token: 5tchbz.ftucc3ugy6g9csu4"
    L48: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L49: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L53: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L54: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L55: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L56: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L57: "cluster.go:117: "
    L58: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L59: "cluster.go:117: "
    L60: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L61: "cluster.go:117: "
    L62: "cluster.go:117:   mkdir -p $HOME/.kube"
    L63: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L64: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L65: "cluster.go:117: "
    L66: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L67: "cluster.go:117: "
    L68: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L69: "cluster.go:117: "
    L70: "cluster.go:117: You should now deploy a pod network to the cluster."
    L71: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L72: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L73: "cluster.go:117: "
    L74: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L75: "cluster.go:117: "
    L76: "cluster.go:117: kubeadm join 10.0.0.139:6443 --token 5tchbz.ftucc3ugy6g9csu4 _"
    L77: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:b3cd9a1bd197d6269dea0c31d1a4b09660ded0108ae43ba39f1c006fc098297b "
    L78: "cluster.go:117: i  Using Cilium version 1.12.1"
    L79: "cluster.go:117: ? Auto-detected cluster name: kubernetes"
    L80: "cluster.go:117: ? Auto-detected datapath mode: tunnel"
    L81: "cluster.go:117: ? Auto-detected kube-proxy has been installed"
    L82: "cluster.go:117: i  helm template --namespace kube-system cilium cilium/cilium --version 1.12.1 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,extraConfig.cluster-pool-ipv4-?cidr=192.168.0.0/17,extraConfig.enable-endpoint-routes=true,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vx?lan"
    L83: "cluster.go:117: i  Storing helm values file in kube-system/cilium-cli-helm-values Secret"
    L84: "cluster.go:117: ? Created CA in secret cilium-ca"
    L85: "cluster.go:117: ? Generating certificates for Hubble..."
    L86: "cluster.go:117: ? Creating Service accounts..."
    L87: "cluster.go:117: ? Creating Cluster roles..."
    L88: "cluster.go:117: ? Creating ConfigMap for Cilium version 1.12.1..."
    L89: "cluster.go:117: i Manual overwrite in ConfigMap: enable-endpoint-routes=true"
    L90: "cluster.go:117: i Manual overwrite in ConfigMap: cluster-pool-ipv4-cidr=192.168.0.0/17"
    L91: "cluster.go:117: ? Creating Agent DaemonSet..."
    L92: "cluster.go:117: ? Creating Operator Deployment..."
    L93: "cluster.go:117: ? Waiting for Cilium to be installed and ready..."
    L94: "cluster.go:117: ? Cilium was successfully installed! Run _cilium status_ to view installation health"
    L95: "cluster.go:117: daemonset.apps/cilium patched"
    L96: "cluster.go:117: ?[33m    /??_"
    L97: "cluster.go:117: ?[36m /???[33m___/?[32m??_?[0m    Cilium:         ?[32mOK?[0m"
    L98: "cluster.go:117: ?[36m ___?[31m/??_?[32m__/?[0m    Operator:       ?[32mOK?[0m"
    L99: "cluster.go:117: ?[32m /???[31m___/?[35m??_?[0m    Hubble:         ?[36mdisabled?[0m"
    L100: "cluster.go:117: ?[32m ___?[34m/??_?[35m__/?[0m    ClusterMesh:    ?[36mdisabled?[0m"
    L101: "cluster.go:117: ?[34m    ___/"
    L102: "cluster.go:117: ?[0m"
    L103: "cluster.go:117: Deployment       cilium-operator    "
    L104: "cluster.go:117: DaemonSet        cilium             "
    L105: "cluster.go:117: Containers:      cilium             "
    L106: "cluster.go:117:                  cilium-operator    "
    L107: "cluster.go:117: Cluster Pods:    0/0 managed by Cilium"
    L108: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L109: "--- FAIL: kubeadm.v1.24.14.cilium.cgroupv1.base/node_readiness (91.68s)"
    L110: "kubeadm.go:295: nodes are not ready: ready nodes should be equal to 2: 1_"
    L111: " "
    L112: "  "

kubeadm.v1.24.14.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.24.14.flannel.cgroupv1.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.25.10.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.25.10.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (3) ❌ Failed: qemu_uefi-arm64 (1, 2)

Diagnostic output for qemu_uefi-arm64, run 2

    L1: " Error: _cluster.go:117: I0907 19:35:55.050853    1453 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.25"
    L2: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.13"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.13"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.13"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.25.13"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.8"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3"
    L9: "cluster.go:117: I0907 19:36:19.168714    1622 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.25"
    L10: "cluster.go:117: [init] Using Kubernetes version: v1.25.13"
    L11: "cluster.go:117: [preflight] Running pre-flight checks"
    L12: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L13: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L14: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L15: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L16: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L17: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L18: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.6?]"
    L19: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L20: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L22: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L27: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L28: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L29: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L30: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L35: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L36: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L37: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L40: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L41: "cluster.go:117: [apiclient] All control plane components are healthy after 7.003756 seconds"
    L42: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L43: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L44: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L45: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L47: "cluster.go:117: [bootstrap-token] Using token: m0vphe.xh6uls32o13worwy"
    L48: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L49: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L53: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L54: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L55: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L56: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L57: "cluster.go:117: "
    L58: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L59: "cluster.go:117: "
    L60: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L61: "cluster.go:117: "
    L62: "cluster.go:117:   mkdir -p $HOME/.kube"
    L63: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L64: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L65: "cluster.go:117: "
    L66: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L67: "cluster.go:117: "
    L68: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L69: "cluster.go:117: "
    L70: "cluster.go:117: You should now deploy a pod network to the cluster."
    L71: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L72: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L73: "cluster.go:117: "
    L74: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L75: "cluster.go:117: "
    L76: "cluster.go:117: kubeadm join 10.0.0.6:6443 --token m0vphe.xh6uls32o13worwy _"
    L77: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:857ea3d1f06eea7fdef142befcac642482b272491cf8a23a9102589961102711 "
    L78: "cluster.go:117: i  Using Cilium version 1.12.1"
    L79: "cluster.go:117: ? Auto-detected cluster name: kubernetes"
    L80: "cluster.go:117: ? Auto-detected datapath mode: tunnel"
    L81: "cluster.go:117: ? Auto-detected kube-proxy has been installed"
    L82: "cluster.go:117: i  helm template --namespace kube-system cilium cilium/cilium --version 1.12.1 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,extraConfig.cluster-pool-ipv4-?cidr=192.168.0.0/17,extraConfig.enable-endpoint-routes=true,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vx?lan"
    L83: "cluster.go:117: i  Storing helm values file in kube-system/cilium-cli-helm-values Secret"
    L84: "cluster.go:117: ? Created CA in secret cilium-ca"
    L85: "cluster.go:117: ? Generating certificates for Hubble..."
    L86: "cluster.go:117: ? Creating Service accounts..."
    L87: "cluster.go:117: ? Creating Cluster roles..."
    L88: "cluster.go:117: ? Creating ConfigMap for Cilium version 1.12.1..."
    L89: "cluster.go:117: i Manual overwrite in ConfigMap: enable-endpoint-routes=true"
    L90: "cluster.go:117: i Manual overwrite in ConfigMap: cluster-pool-ipv4-cidr=192.168.0.0/17"
    L91: "cluster.go:117: ? Creating Agent DaemonSet..."
    L92: "cluster.go:117: ? Creating Operator Deployment..."
    L93: "cluster.go:117: ? Waiting for Cilium to be installed and ready..."
    L94: "cluster.go:117: ? Cilium was successfully installed! Run _cilium status_ to view installation health"
    L95: "cluster.go:117: daemonset.apps/cilium patched"
    L96: "cluster.go:117: ?[33m    /??_"
    L97: "cluster.go:117: ?[36m /???[33m___/?[32m??_?[0m    Cilium:         ?[32mOK?[0m"
    L98: "cluster.go:117: ?[36m ___?[31m/??_?[32m__/?[0m    Operator:       ?[32mOK?[0m"
    L99: "cluster.go:117: ?[32m /???[31m___/?[35m??_?[0m    Hubble:         ?[36mdisabled?[0m"
    L100: "cluster.go:117: ?[32m ___?[34m/??_?[35m__/?[0m    ClusterMesh:    ?[36mdisabled?[0m"
    L101: "cluster.go:117: ?[34m    ___/"
    L102: "cluster.go:117: ?[0m"
    L103: "cluster.go:117: Deployment       cilium-operator    "
    L104: "cluster.go:117: DaemonSet        cilium             "
    L105: "cluster.go:117: Containers:      cilium             "
    L106: "cluster.go:117:                  cilium-operator    "
    L107: "cluster.go:117: Cluster Pods:    0/0 managed by Cilium"
    L108: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L109: "harness.go:582: Found emergency shell on machine 48826a2a-bbc9-4b50-bde8-6de7062dd7fd console"
    L110: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine 48826a2a-bbc9-4b50-bde8-6de7062dd7fd console"
    L111: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39migni???te.target?[0m - Ignition Complete. ) on machine 48826a2a-bbc9-4b50-bde8-6de7062dd7fd console_"
    L112: " "

Diagnostic output for qemu_uefi-arm64, run 1

    L1: "  "
    L2: " Error: _cluster.go:117: I0907 19:22:38.628858    1433 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.25"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.13"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.13"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.13"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.25.13"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.8"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L9: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3"
    L10: "cluster.go:117: I0907 19:22:56.504418    1595 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.25"
    L11: "cluster.go:117: [init] Using Kubernetes version: v1.25.13"
    L12: "cluster.go:117: [preflight] Running pre-flight checks"
    L13: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L14: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L15: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L16: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L17: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L18: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L19: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.8?9]"
    L20: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L22: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L27: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L28: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L29: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L30: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L35: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L36: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L37: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L40: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L41: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L42: "cluster.go:117: [apiclient] All control plane components are healthy after 6.504924 seconds"
    L43: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L44: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L45: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L47: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L48: "cluster.go:117: [bootstrap-token] Using token: tz2ry0.7dz7qqwijml1r8ly"
    L49: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L53: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L54: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L55: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L56: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L57: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L58: "cluster.go:117: "
    L59: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L60: "cluster.go:117: "
    L61: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L62: "cluster.go:117: "
    L63: "cluster.go:117:   mkdir -p $HOME/.kube"
    L64: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L65: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L66: "cluster.go:117: "
    L67: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L68: "cluster.go:117: "
    L69: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L70: "cluster.go:117: "
    L71: "cluster.go:117: You should now deploy a pod network to the cluster."
    L72: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L73: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L74: "cluster.go:117: "
    L75: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L76: "cluster.go:117: "
    L77: "cluster.go:117: kubeadm join 10.0.0.89:6443 --token tz2ry0.7dz7qqwijml1r8ly _"
    L78: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:cfc0e6fc1e336e6c5f94e020cb6923aaf97ebce2e75f5d5ac25980927029f55e "
    L79: "cluster.go:117: i  Using Cilium version 1.12.1"
    L80: "cluster.go:117: ? Auto-detected cluster name: kubernetes"
    L81: "cluster.go:117: ? Auto-detected datapath mode: tunnel"
    L82: "cluster.go:117: ? Auto-detected kube-proxy has been installed"
    L83: "cluster.go:117: i  helm template --namespace kube-system cilium cilium/cilium --version 1.12.1 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,extraConfig.cluster-pool-ipv4-?cidr=192.168.0.0/17,extraConfig.enable-endpoint-routes=true,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vx?lan"
    L84: "cluster.go:117: i  Storing helm values file in kube-system/cilium-cli-helm-values Secret"
    L85: "cluster.go:117: ? Created CA in secret cilium-ca"
    L86: "cluster.go:117: ? Generating certificates for Hubble..."
    L87: "cluster.go:117: ? Creating Service accounts..."
    L88: "cluster.go:117: ? Creating Cluster roles..."
    L89: "cluster.go:117: ? Creating ConfigMap for Cilium version 1.12.1..."
    L90: "cluster.go:117: i Manual overwrite in ConfigMap: enable-endpoint-routes=true"
    L91: "cluster.go:117: i Manual overwrite in ConfigMap: cluster-pool-ipv4-cidr=192.168.0.0/17"
    L92: "cluster.go:117: ? Creating Agent DaemonSet..."
    L93: "cluster.go:117: ? Creating Operator Deployment..."
    L94: "cluster.go:117: ? Waiting for Cilium to be installed and ready..."
    L95: "cluster.go:117: ? Cilium was successfully installed! Run _cilium status_ to view installation health"
    L96: "cluster.go:117: daemonset.apps/cilium patched"
    L97: "cluster.go:117: ?[33m    /??_"
    L98: "cluster.go:117: ?[36m /???[33m___/?[32m??_?[0m    Cilium:         ?[32mOK?[0m"
    L99: "cluster.go:117: ?[36m ___?[31m/??_?[32m__/?[0m    Operator:       ?[32mOK?[0m"
    L100: "cluster.go:117: ?[32m /???[31m___/?[35m??_?[0m    Hubble:         ?[36mdisabled?[0m"
    L101: "cluster.go:117: ?[32m ___?[34m/??_?[35m__/?[0m    ClusterMesh:    ?[36mdisabled?[0m"
    L102: "cluster.go:117: ?[34m    ___/"
    L103: "cluster.go:117: ?[0m"
    L104: "cluster.go:117: Deployment       cilium-operator    "
    L105: "cluster.go:117: DaemonSet        cilium             "
    L106: "cluster.go:117: Containers:      cilium             "
    L107: "cluster.go:117:                  cilium-operator    "
    L108: "cluster.go:117: Cluster Pods:    0/0 managed by Cilium"
    L109: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L110: "harness.go:582: Found emergency shell on machine 01b4011a-a336-435b-a819-ebccdab70544 console"
    L111: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine 01b4011a-a336-435b-a819-ebccdab70544 console"
    L112: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39migni???te.target?[0m - Ignition Complete. ) on machine 01b4011a-a336-435b-a819-ebccdab70544 console_"
    L113: " "

kubeadm.v1.25.10.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.26.5.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.26.5.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.26.5.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.27.2.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.27.2.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.27.2.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (2) ❌ Failed: qemu_uefi-arm64 (1)

Diagnostic output for qemu_uefi-arm64, run 1

    L1: "  "
    L2: " Error: _cluster.go:117: I0907 19:26:21.196493    1455 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.27"
    L3: "cluster.go:117: W0907 19:26:21.294098    1455 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.5, falling back to the nearest etcd version (3.5.7-0)"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.27.5"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.27.5"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.27.5"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.27.5"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.9"
    L9: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.7-0"
    L10: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.10.1"
    L11: "cluster.go:117: I0907 19:26:38.216496    1615 version.go:256] remote version is much newer: v1.28.1; falling back to: stable-1.27"
    L12: "cluster.go:117: [init] Using Kubernetes version: v1.27.5"
    L13: "cluster.go:117: [preflight] Running pre-flight checks"
    L14: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L15: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L16: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L17: "cluster.go:117: W0907 19:26:38.520831    1615 checks.go:835] detected that the sandbox image __registry.k8s.io/pause:3.8__ of the container runtime is inconsistent with that used by kubeadm. It is rec?ommended that using __registry.k8s.io/pause:3.9__ as the CRI sandbox image."
    L18: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L19: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L20: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L21: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.1?18]"
    L22: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L23: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L24: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L27: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L28: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L29: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L30: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L31: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L32: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L34: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L35: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L36: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L37: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L38: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L39: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L40: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L41: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L42: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L43: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L44: "cluster.go:117: [apiclient] All control plane components are healthy after 6.003313 seconds"
    L45: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L46: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L47: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L48: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L49: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L50: "cluster.go:117: [bootstrap-token] Using token: wo0rdx.fc7oocc7dt4h0d7p"
    L51: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L53: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L54: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L55: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L56: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L57: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L58: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L59: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L60: "cluster.go:117: "
    L61: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L62: "cluster.go:117: "
    L63: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L64: "cluster.go:117: "
    L65: "cluster.go:117:   mkdir -p $HOME/.kube"
    L66: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L67: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L68: "cluster.go:117: "
    L69: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L70: "cluster.go:117: "
    L71: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L72: "cluster.go:117: "
    L73: "cluster.go:117: You should now deploy a pod network to the cluster."
    L74: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L75: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L76: "cluster.go:117: "
    L77: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L78: "cluster.go:117: "
    L79: "cluster.go:117: kubeadm join 10.0.0.118:6443 --token wo0rdx.fc7oocc7dt4h0d7p _"
    L80: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:36e58c30aa3e52c96d7419fc7c4b3e98ff9a6efe377c3f2bfaf1f7eb541e9e6e "
    L81: "cluster.go:117: namespace/kube-flannel created"
    L82: "cluster.go:117: clusterrole.rbac.authorization.k8s.io/flannel created"
    L83: "cluster.go:117: clusterrolebinding.rbac.authorization.k8s.io/flannel created"
    L84: "cluster.go:117: serviceaccount/flannel created"
    L85: "cluster.go:117: configmap/kube-flannel-cfg created"
    L86: "cluster.go:117: daemonset.apps/kube-flannel-ds created"
    L87: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L88: "harness.go:582: Found emergency shell on machine fa4d73d5-12e4-4184-94c7-8f4f6dce3f37 console"
    L89: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine fa4d73d5-12e4-4184-94c7-8f4f6dce3f37 console"
    L90: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39mtorc???te torcx store to satisfy profile. ) on machine fa4d73d5-12e4-4184-94c7-8f4f6dce3f37 console_"
    L91: " "

linux.nfs.v3 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

linux.nfs.v4 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

linux.ntp 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

packages 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

skipped qemu update tests 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

systemd.journal.remote 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.journal.user 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.sysext.custom-docker 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.sysext.custom-oem 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.sysext.simple 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.sysusers.gshadow 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

torcx.enable-service 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

krnowak · 2023-09-08T09:47:49Z

sdk_container/src/third_party/coreos-overlay/coreos-base/misc-files/files/oems/azure

@@ -0,0 +1,13 @@
+/etc/systemd/system/nvidia.service


I assume that the deletion happens after /etc is already overlaid? Also hopefully this won't delete third party files that accidentally had the same name.

Yes, after the overlay. For user files with the same service name that we bring in I think the most practical solution is to document this because the same service name used will have other issues, too, I will add it to the changelog.

The vendor tools on the OEM partition weren't updated. We now want to ship them as systemd-sysext images which we can easily update. This change extends the Flatcar A/B update mechanism to cover the OEM systemd-sysext images. The same mechanism is also able to support "official" Flatcar extensions, e.g., a ZFS extension.

The kola run didn't pick up the version that was set up in the build because the git changes from that step are lost. Redo the version setup in the kola run to use the same version, and skip the kola update test if no update payload can be found. In the future we should copy it over from the GitHub Action artifact.

pothos had a problem deploying to development July 26, 2023 11:22 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 2f80c38 to 76f88cc Compare July 26, 2023 11:35

pothos had a problem deploying to development July 26, 2023 11:35 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 76f88cc to 04d742c Compare July 26, 2023 15:27

pothos had a problem deploying to development July 26, 2023 15:27 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 04d742c to 3228c2a Compare July 27, 2023 13:53

pothos had a problem deploying to development July 27, 2023 13:53 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 3228c2a to b041d6b Compare July 28, 2023 16:29

pothos had a problem deploying to development July 28, 2023 16:29 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from b041d6b to 22cb99d Compare July 28, 2023 17:12

pothos had a problem deploying to development July 28, 2023 17:12 — with GitHub Actions Failure

pothos mentioned this pull request Jul 28, 2023

Support OEM systemd-sysext images and Flatcar extensions flatcar/update_engine#24

Merged

pothos force-pushed the kai/update-sysext branch from 22cb99d to fe03ba3 Compare August 30, 2023 16:51

pothos had a problem deploying to development August 30, 2023 16:51 — with GitHub Actions Error

pothos mentioned this pull request Aug 31, 2023

sysext: port AWS OEM to systemd sysext image #1083

Merged

4 tasks

pothos force-pushed the kai/update-sysext branch from fe03ba3 to 6a94a91 Compare September 1, 2023 08:40

pothos had a problem deploying to development September 1, 2023 08:40 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 6a94a91 to 82bf3ad Compare September 1, 2023 08:57

pothos had a problem deploying to development September 1, 2023 08:58 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 82bf3ad to 50905a0 Compare September 1, 2023 09:42

pothos had a problem deploying to development September 1, 2023 09:42 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 50905a0 to d5daa5a Compare September 1, 2023 12:50

pothos had a problem deploying to development September 1, 2023 12:50 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from d5daa5a to c40f3c9 Compare September 4, 2023 14:03

pothos had a problem deploying to development September 4, 2023 14:03 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from c40f3c9 to 46441af Compare September 4, 2023 16:29

pothos had a problem deploying to development September 4, 2023 16:29 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 46441af to fd62ad5 Compare September 4, 2023 16:40

pothos had a problem deploying to development September 4, 2023 16:40 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from fd62ad5 to ce198bf Compare September 4, 2023 19:41

pothos had a problem deploying to development September 4, 2023 19:42 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from ce198bf to 67e5023 Compare September 4, 2023 19:55

pothos had a problem deploying to development September 4, 2023 19:55 — with GitHub Actions Error

pothos mentioned this pull request Sep 4, 2023

kola: Add new OEM sysext update and migration test flatcar/mantle#448

Merged

pothos force-pushed the kai/update-sysext branch from c7bc20a to aec5322 Compare September 6, 2023 09:31

pothos changed the title ~~wip: Support OEM systemd-sysext images and Flatcar extensions~~ Support OEM systemd-sysext images and Flatcar extensions Sep 6, 2023

pothos force-pushed the kai/update-sysext branch 2 times, most recently from 50781e7 to 7d66a67 Compare September 6, 2023 09:48

pothos temporarily deployed to development September 6, 2023 09:48 — with GitHub Actions Inactive

tormath1 reviewed Sep 6, 2023

View reviewed changes

sdk_container/src/third_party/coreos-overlay/coreos-base/misc-files/misc-files-0.ebuild Outdated Show resolved Hide resolved

pothos had a problem deploying to development September 6, 2023 10:53 — with GitHub Actions Error

pothos force-pushed the kai/update-sysext branch from 718bf7f to d72d746 Compare September 6, 2023 10:53

pothos temporarily deployed to development September 6, 2023 10:53 — with GitHub Actions Inactive

pothos temporarily deployed to development September 6, 2023 15:47 — with GitHub Actions Inactive

pothos force-pushed the kai/update-sysext branch from 22e1ee8 to 7b25b11 Compare September 6, 2023 16:28

pothos temporarily deployed to development September 6, 2023 16:29 — with GitHub Actions Inactive

pothos force-pushed the kai/update-sysext branch from 7b25b11 to 1370eb2 Compare September 7, 2023 11:47

pothos temporarily deployed to development September 7, 2023 11:47 — with GitHub Actions Inactive

pothos force-pushed the kai/update-sysext branch from 1370eb2 to e3aa677 Compare September 7, 2023 15:50

pothos temporarily deployed to development September 7, 2023 15:50 — with GitHub Actions Inactive

krnowak approved these changes Sep 8, 2023

View reviewed changes

pothos added 2 commits September 8, 2023 14:50

pothos force-pushed the kai/update-sysext branch from e3aa677 to ddb808f Compare September 8, 2023 12:51

pothos had a problem deploying to development September 8, 2023 12:51 — with GitHub Actions Failure

pothos merged commit 6bec600 into main Sep 8, 2023
1 check failed

pothos deleted the kai/update-sysext branch September 8, 2023 12:53

github-actions bot mentioned this pull request Nov 9, 2023

Monthly contributions report 2023-08-22 - 2023-09-21 flatcar/Flatcar#1245

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support OEM systemd-sysext images and Flatcar extensions #1016

Support OEM systemd-sysext images and Flatcar extensions #1016

pothos commented Jul 26, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023 •

edited

Loading

krnowak Sep 8, 2023

pothos Sep 8, 2023

Support OEM systemd-sysext images and Flatcar extensions #1016

Support OEM systemd-sysext images and Flatcar extensions #1016

Conversation

pothos commented Jul 26, 2023 • edited Loading

How to use

Testing done

github-actions bot commented Sep 6, 2023 • edited Loading

Test report for 3718.0.0+nightly-20230905-2100 / amd64 arm64

krnowak Sep 8, 2023

Choose a reason for hiding this comment

pothos Sep 8, 2023

Choose a reason for hiding this comment

pothos commented Jul 26, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023 •

edited

Loading