kubeadm doesn't pass etcd progress notification flag

[p0lyn0mial]

What happened?

The experimental-watch-progress-notify-interval flag specifies an interval at which etcd sends data to the kube-api server.

It is used by the WatchBookmark feature ( kube-apiserver) which is GA since 1.17.
It will also be used by a new WatchList feature (kube-apiserver's) which is Alpha since 1.25

Given the above, a default installation must pass the flag required by the kube-apiserver to etcd. Otherwise, some features will not work.

Note that the feature was graduated to GA (non-experiment) in etcd 3.5 without any code changes. Once kubernetes switches to etcd in 3.5 the flag name will have to be changed.

What did you expect to happen?

The experimental-watch-progress-notify-interval flag is specified on etcd binary.

How can we reproduce it (as minimally and precisely as possible)?

atm the kubeadm does not specify the flag. So simply installing a cluster is enough to reproduce the issue.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

n/a

Cloud provider

n/a

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

‐---------

edit neolit123

1.25

• add the experimental flag to kubeadm
  kubeadm: pass etcd progress notification flag to etcd kubeadm: pass etcd progress notification flag to etcd kubernetes#111383

next

• remove experimental flag and use graduated flag
  Track stabilization of --experimental-watch-progress-notify-interval flag etcd-io/etcd#13779

[k8s-ci-robot]

@p0lyn0mial: There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@p0lyn0mial: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[p0lyn0mial]

/cc @neolit123 @aojea

[k8s-ci-robot]

@neolit123: Something went wrong or the destination repo kubernetes/khbeadm does not exist.

In response to this:

/transfer khbeadm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/transfer kubeadm

[neolit123]

/kind feature
/remove-kind bug

[wojtek-t]

FWIW - I'm copying my comment from the original issue to give a context:

1. The feature was graduated to GA (non-experiment) in etcd 3.5 without any code changes so I think we're fine.

2. We already enabled that for all GCE tests long time ago:
   https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-test.sh#L578
   https://github.com/kubernetes/kubernetes/blob/8415ae647d2c433c89910a0e677094e3a20ffb2b/cluster/gce/gci/configure-helper.sh#L1851

3. We're heavily relying on this setting for efficient watch resumption KEP:
   https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/1904-efficient-watch-resumption/README.md#proposal
   [which is already GA]

4. FWIW - we enabled that setting in GKE (together with etcd 3.4) quite some time ago

So I don't consider this an experimental feature. The problem is that there weren't any etcd release after 3.4 for ~2 years, and 3.5 that was released last year is still not fully production ready (not qualified enough).

So I personally think that we are now ready to just enable it in kind for everyone, but I'm not kind maintainer so it needs to be decided by kind maintainers.

[pacoxu]

Link the etcd issue for graduating this flag: etcd-io/etcd#13779

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pacoxu]

/remove-lifecycle stale

[pacoxu]

This is in the plan of etcd v3.6. Move it to v1.29 then.

[liangyuanpeng]

/sig etcd

IMO

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale