✨ Add bootCommands to cloud-init file generation #11271

davidumea · 2024-10-08T14:17:23Z

What this PR does / why we need it:
Adds the ability to provide bootcmd commands to cloud-init via the KubeadmConfig or KubeadmConfigTemplates custom resources. (Same for KubeadmControlPlane)

One thing that I know that does not work at the time of creating this PR is validating/mutating webhooks for KubeadmConfig and KubeadmConfigTemplates. Some pointers to where to change that would be appreciated.

Which issue(s) this PR fixes
Fixes #

k8s-ci-robot · 2024-10-08T14:17:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-10-08T14:17:32Z

Welcome @davidumea!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-10-08T14:17:33Z

Hi @davidumea. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-10-08T14:21:48Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

davidumea · 2024-10-09T12:28:47Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

Exactly I don't think that should be an issue as bootcmd has been part of cloud-init for a long time.

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

During my testing, if the bootcmd module fails e.g. due to invalid commands, cloud-init still continues with the following modules. I haven't been able to test running this with a cloud-init version that doesn't support bootcmd.

davidumea · 2024-10-17T12:30:16Z

@chrischdi Is there anything I can do to get this reviewed?

fabriziopandini · 2024-10-18T08:37:00Z

/ok-to-test

cc @enxebre @vincepri @randomvariable

randomvariable

Bootcmd module has been in cloud-init for as long as I can remember (which is back to 2010), so don't see an issue there.

I would like the documentation to be more explicit though, particularly about how this will have no effect on Ignition. Have suggested some changes.

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go

… PostKubeadmCommands

neolit123 · 2024-10-18T15:05:09Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


shouldn't it be added only to v1beta1 which is the actively maintained API version?

If I remove this and run make verify, it removes bootCommands from the CRDs.

Might be an issue in the verify logic, what do you think?

diff --git a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml index 79c3b19e0..4e1852f77 100644 --- a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml +++ b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml @@ -47,17 +47,6 @@ spec: KubeadmConfigSpec defines the desired state of KubeadmConfig. Either ClusterConfiguration and InitConfiguration should be defined or the JoinConfiguration should be defined. properties: - bootCommands: - description: |- - BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd - module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition. - items: - description: BootCommand defines input for each bootcmd command - in cloud-init. - items: - type: string - type: array - type: array clusterConfiguration: description: ClusterConfiguration along with InitConfiguration are the configurations necessary for the init command

etc

New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

fabriziopandini · 2024-10-21T08:36:03Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

fabriziopandini · 2024-10-21T08:37:14Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -34,6 +34,7 @@ func TestNewInitControlPlaneAdditionalFileEncodings(t *testing.T) {
 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
 			Header:              "test",
+			BootCommands:        nil,


Can we have a test for BootCommands not nil for init, join CP, join workers?

fabriziopandini · 2024-10-21T08:39:02Z

controlplane/kubeadm/internal/filters_test.go

@@ -548,7 +548,7 @@ func TestMatchInitOrJoinConfiguration(t *testing.T) {
    },
    JoinConfiguration: nil,
    Files:             nil,
-    ... // 10 identical fields
+    ... // 11 identical fields


I need to take some time to dig into impact of this change on existing clusters, because we need to make sure this change does not trigger rollouts
However, I'm not sure when I will get to it 😓

Add bootCommands cloud-init file generation

7680f8e

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Oct 8, 2024

k8s-ci-robot requested a review from elmiko October 8, 2024 14:17

k8s-ci-robot requested a review from enxebre October 8, 2024 14:17

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 8, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 18, 2024

fixup: go lint

d5f109b

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 18, 2024

davidumea added 3 commits October 18, 2024 11:39

fixup: lint: added missing newline

2995aad

fixup: add changes from verify-gen

bd016d4

fixup: filter tests

48373f0

randomvariable suggested changes Oct 18, 2024

View reviewed changes

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

fixup: clearer documentation for BootCommands, PreKubeadmCommands and…

4693de5

… PostKubeadmCommands

davidumea requested a review from randomvariable October 18, 2024 13:45

fixup: verify: reflect new documentation in the crds

ebc8a7c

fabriziopandini added the area/provider/bootstrap-kubeadm Issues or PRs related to CAPBK label Oct 18, 2024

k8s-ci-robot removed the do-not-merge/needs-area PR is missing an area label label Oct 18, 2024

neolit123 reviewed Oct 18, 2024

View reviewed changes

fabriziopandini reviewed Oct 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add bootCommands to cloud-init file generation #11271

✨ Add bootCommands to cloud-init file generation #11271

davidumea commented Oct 8, 2024 •

edited

Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

neolit123 Oct 18, 2024

davidumea Oct 21, 2024

fabriziopandini Oct 21, 2024

fabriziopandini Oct 21, 2024

fabriziopandini Oct 21, 2024

fabriziopandini Oct 21, 2024

	dst.Files = restored.Files

	dst.Users = restored.Users
	if restored.Users != nil {
	for i := range restored.Users {
	if restored.Users[i].PasswdFrom != nil {
	dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom
	}
	}
	}

✨ Add bootCommands to cloud-init file generation #11271

Are you sure you want to change the base?

✨ Add bootCommands to cloud-init file generation #11271

Conversation

davidumea commented Oct 8, 2024 • edited Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 • edited Loading

davidumea commented Oct 9, 2024 • edited Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

Choose a reason for hiding this comment

neolit123 Oct 18, 2024

Choose a reason for hiding this comment

davidumea Oct 21, 2024

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

davidumea commented Oct 8, 2024 •

edited

Loading

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading