📖 Proposal: MachineDrainRules

[sbueringer]

Signed-off-by: Stefan Büringer buringerst@vmware.com

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Part of #11240

[sbueringer]

/assign @chrischdi @enxebre @fabriziopandini @killianmuldoon @vincepri @JoelSpeed

[kreeuwijk]

Loving this solution for CAPI-11024, great work!

[sftim]

For more on the label vs. annotation thinking, see kubernetes/kubernetes#127247 (comment)

[sftim]

In general in CAPI I think we can only define annotations in our namespace.

Kubernetes is open source; anyone can propose registering an annotation (all it needs is a PR against https://kubernetes.io/docs/reference/labels-annotations-taints/)

[sbueringer]

In general in CAPI I think we can only define annotations in our namespace.

Kubernetes is open source; anyone can propose registering an annotation (all it needs is a PR against https://kubernetes.io/docs/reference/labels-annotations-taints/)

I'm aware it's open source 😀. I assumed that these would typically be defined through KEPs

[kreeuwijk]

There is one more use case that we've seen with a customer, I was wondering if this could be facilitated by MachineDrainRules as well. We had a customer that ran gaming sessions (Minecraft) in Kubernetes, so a pod would represent an active game in progress. When the customer wanted to e.g. upgrade Kubernetes, they needed to change the upgrade strategy in the MachineDeployment from RollingUpdate to OnDelete, to prevent CAPI from just draining a node at random, killing the game sessions that were in progress on that node. With OnDelete, they could manually cordon a node, wait for game sessions on it to naturally end and then manually delete the Machine object to deprovision the node. CAPI would then install a fresh one. The customer had to go through all their 100 nodes this way, which was not a fun exercise.

If MachineDrainRules supported something like a "WaitForCompletion" mode of draining (for pods matching certain labels), a customer like this could configure a rule that made CAPI wait until pods matching the "wait" labels are naturally terminated. This would enable a node drain to function as follows:

• Cordon the node so it doesn't retrieve new workloads
• Optionally drain certain pods that are allowed to be evicted at this time (based on priority)
• Wait until pods matching the "wait" labels are naturally terminated
• Drain remaining pods (based on priority)
• Drain complete, node can be deprovisioned.

While this could make repaves take a long time when such a wait rule is configured, it is still much preferable over having to manage the drains entirely manually. There could even be a wait timeout setting that sets a maximum amount of time that the system will wait for such pods to terminate naturally, before draining them anyway.

[chrischdi]

There is one more use case that we've seen with a customer, I was wondering if this could be facilitated by MachineDrainRules as well. We had a customer that ran gaming sessions (Minecraft) in Kubernetes, so a pod would represent an active game in progress. When the customer wanted to e.g. upgrade Kubernetes, they needed to change the upgrade strategy in the MachineDeployment from RollingUpdate to OnDelete, to prevent CAPI from just draining a node at random, killing the game sessions that were in progress on that node. With OnDelete, they could manually cordon a node, wait for game sessions on it to naturally end and then manually delete the Machine object to deprovision the node. CAPI would then install a fresh one. The customer had to go through all their 100 nodes this way, which was not a fun exercise.

If MachineDrainRules supported something like a "WaitForCompletion" mode of draining (for pods matching certain labels), a customer like this could configure a rule that made CAPI wait until pods matching the "wait" labels are naturally terminated. This would enable a node drain to function as follows:

• Cordon the node so it doesn't retrieve new workloads
• Optionally drain certain pods that are allowed to be evicted at this time (based on priority)
• Wait until pods matching the "wait" labels are naturally terminated
• Drain remaining pods (based on priority)
• Drain complete, node can be deprovisioned.

While this could make repaves take a long time when such a wait rule is configured, it is still much preferable over having to manage the drains entirely manually. There could even be a wait timeout setting that sets a maximum amount of time that the system will wait for such pods to terminate naturally, before draining them anyway.

Doesn't this already work for upgrades?

• Start the upgrade
• The feature to cordon nodes which are outdated (CAPI controller should taint outdated nodes with PreferNoSchedule #7043) will add a taint so new pods should get scheduled on new nodes
• Drain gets done, use PDB's to prevent eviction of "gaming session" pods, other's get evicted already.
• Drain succeeds when the pods are completed (because they don't need to get evicted anymore)

[kreeuwijk]

I suppose using PDBs would indeed work to make the drain process wait for the pods to terminate naturally. Thanks for the tip 👍🏻

[sbueringer]

All findings should be addressed via aa3ea2b

PTAL :)

[fabriziopandini]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: c816aab3eef91dbcb5aa0b278bf841bf06e5fa66

[fabriziopandini]

As per Oct 9th office hours, lazy consensus deadline set for next Friday (Oct 18th)

[enxebre]

/lgtm

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.

[JoelSpeed]

/lgtm

[fabriziopandini]

Great to see we reached consensus for this first iteration!
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [fabriziopandini]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment