KEP: Support Instance Metadata Service with Cloud Controller Manager #1158
Conversation
k8s-ci-robot
added
the
sig/cloud-provider
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
the
kind/kep
|
/assign @cheftako |
|
@khenidak @andrewsykim @justinsb addressed comments. PTAL. |
|
In theory you could do this today by:
The only caveat is that you can't sync the node initialization taint with the DaemonSet. #1003 would fix that though. But, you could register kubelet with some provider-specific taint and untaint that from the DaemonSet after the node is updated. |
| With these changes, the whole node initialization workflow would be: | ||
|
|
||
| - Kubelet specifying `--cloud-provider=external` will add a taint `node.cloudprovider.kubernetes.io/uninitialized` with an effect NoSchedule during initialization. | ||
| - `cloud-node-manager` would initialize the node again with `Instances` and `Zones`. |
There was a problem hiding this comment.
how would this be deployed? Does it need any special cases on Linux/Windows?
There was a problem hiding this comment.
it is running as a daemonset on Linux nodes, I think it can be also running as the daemonset on Windows.
|
@andrewsykim @feiskyer -- what are next steps here? |
|
I asked @feiskyer to demo this in the SIG meeting but it's been hard to schedule this given the timezone difference. I think we want to keep this option open for cloud providers that run into quota issues but we're not making this the standard for all providers. |
|
@andrewsykim -- Maybe @feiskyer can record a demo and post it here and to the mailing list for discussion? |
|
That sounds good to me |
|
I won't be at KubeCON this time around but I'm sure @cheftako @nckturner @yastij would be interested |
|
@andrewsykim @cheftako @nckturner @yastij @justaugustus Marked the KEP as implementable as we talked about on KubeCon NA, PTAL |
|
kindly ping @andrewsykim @cheftako @nckturner @yastij @justaugustus |
|
Sorry for chiming it so late, I didn't see this KEP previously. |
|
The KEP is not coupled with IPv4, it aims to support IMDS to reduce number of cloud provider API calls. It also works for IPv6 if the cloud provider supports IPv6 for IMDS. |
|
Pinged in Slack to raise reviewers: https://kubernetes.slack.com/archives/C718BPBQ8/p1578497779005800 |
k8s-ci-robot
added
approved
|
The KEP is now moved to provider Azure /area provider/azure |
|
I think keeping this in scope for Azure makes sense, we should share this with other subprojects/providers to see if we should make this generally available for everyone. I added @feiskyer's demo video (https://www.youtube.com/watch?v=47rK-xLG_88) in the next SIG agenda meeting to see if other providers are interested in this. |
|
@feiskyer -- Will review next week. |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andyzhangx, feiskyer The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@aoxn this sounds like it would be useful to you based on the API rate limiting issues you've brought up. |
|
@andrewsykim Hi, thanks for the notification. I do have a question, A cloud-node-manager daemonset means that 5k Concurrent CMDS API(NodeAddress reconcile) call would be issued to CloudAPI in a cluster with 5k nodes? Is this would be a new throttle in API call? Actually, we already face this issue in some daemonset in large scale cluster. |
|
The cloud-node-manager daemonset would reconcile using instance metadata though (usually separate from cloud API). |
|
@andrewsykim Is there any chances that instance metadata might be throttled ? As far as i know, alibaba cloud `s instance metadata is another kind of Center API System which might have its own rate limit. How about azure? @feiskyer |
Not sure, it depends on the Cloud implementations. But even there're possibilities, large throughput should be allowed compared to Cloud API. |
KEP: Support Instance Metadata Service with Cloud Controller Manager.
Refer kubernetes/cloud-provider#30.
/sig cloud-provider
/assign @andrewsykim @craiglpeters @justaugustus