Fix timeout when defragmenting etcd on startup #11164

brandond · 2024-10-24T00:19:03Z

Proposed Changes

Fix timeout when defragmenting etcd on startup:

Use custom client with no hidden background connection, and without automatic retry of RPCs.
Only timeout status requests; allow defrag and alarm clear requests to run to completion. The defrag and alarm clear may take more than 30 seconds to complete if the datastore is large and the disk is slow.
Don't clear all alarms on the cluster after defragging, just clear the NOSPACE alarm on the local node.
Print datastore utilization before/after defrag
Add tests

Types of Changes

bugfix / enhancement

Verification

See linked issue - however this seems to require a large (8gb+) fragmented datastore on a fairly slow disk to reproduce.

Testing

new tests added

Linked Issues

User-Facing Change

Further Comments

brandond · 2024-10-24T00:20:41Z

pkg/etcd/etcd.go

@@ -1593,24 +1598,6 @@ func (e *ETCD) GetMembersClientURLs(ctx context.Context) ([]string, error) {
 	return clientURLs, nil
 }

-// GetMembersNames will list through the member lists in etcd and return
-// back a combined list of member names
-func (e *ETCD) GetMembersNames(ctx context.Context) ([]string, error) {


Function was not used here or in rke2 and is not required by the interface.

codecov · 2024-10-24T00:53:02Z

Codecov Report

Attention: Patch coverage is 46.82540% with 67 lines in your changes missing coverage. Please review.

Project coverage is 38.10%. Comparing base (c0d661b) to head (09cea43).
Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/etcd/etcd.go	48.05%	32 Missing and 8 partials ⚠️
pkg/etcd/resolver.go	45.83%	23 Missing and 3 partials ⚠️
pkg/etcd/snapshot.go	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (c0d661b) and HEAD (09cea43). Click for more details.

HEAD has 3 uploads less than BASE

Flag BASE (c0d661b) HEAD (09cea43)

unittests 1 0

e2etests 8 6

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #11164       +/-   ##
===========================================
- Coverage   48.72%   38.10%   -10.63%     
===========================================
  Files         178      162       -16     
  Lines       14813    17979     +3166     
===========================================
- Hits         7218     6850      -368     
- Misses       6288     9932     +3644     
+ Partials     1307     1197      -110

Flag	Coverage Δ
e2etests	`33.97% <0.00%> (-12.10%)`	⬇️
inttests	`34.72% <46.82%> (?)`
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

manuelbuil · 2024-10-24T07:31:57Z

Shouldn't we add a (larger) timeout for defrag? Could etcd get stuck in defrag forever for some reason?

brandond · 2024-10-24T08:07:32Z

The reason we added a timeout on the status command is that the etcd grpc client will retry it automatically if the connection to the service is unavailable. I don't believe the other commands are subject to retry, but I'm going to add some tests to this to verify behavior before merging.

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

brandond · 2024-10-25T23:18:21Z

Confirmed no breakage here:

NAME           STATUS   ROLES                       AGE     VERSION                INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                   KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-1   Ready    control-plane,etcd,master   2m33s   v1.31.1+k3s-5e6dc725   172.17.0.4    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1
k3s-server-2   Ready    etcd                        80s     v1.31.1+k3s-5e6dc725   172.17.0.5    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1
k3s-server-3   Ready    control-plane,etcd,master   58s     v1.31.1+k3s-5e6dc725   172.17.0.6    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1
k3s-server-4   Ready    control-plane,master        33s     v1.31.1+k3s-5e6dc725   172.17.0.7    <none>        K3s v1.31.1-k3s-5e6dc725   6.8.0-1016-aws   containerd://1.7.22-k3s1

Confirmed no breakage on the RKE2 side with these changes: rancher/rke2#7139

And the startup actually seems to be a bit faster, with less log spam while etcd is starting.

* Use clientv3.NewCtxClient instead of New to avoid automatic retry of all RPCs * Only timeout status requests; allow defrag and alarm clear requests to run to completion. * Only clear alarms on the local cluster member, not ALL cluster members Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

pkg/etcd/etcd.go

fmoral2 · 2024-11-11T16:10:28Z

pkg/etcd/etcd_test.go

+			wantErr: false,
+		},
+		{
+			name: "slow defrag",


brandond requested a review from a team as a code owner October 24, 2024 00:19

brandond commented Oct 24, 2024

View reviewed changes

brandond force-pushed the etcd-defrag-no-timeout branch from 796d8a9 to a3d1693 Compare October 25, 2024 08:01

dereknola previously approved these changes Oct 25, 2024

View reviewed changes

cwayne18 previously approved these changes Oct 25, 2024

View reviewed changes

brandond dismissed stale reviews from cwayne18 and dereknola via cae9e2b October 25, 2024 21:35

brandond force-pushed the etcd-defrag-no-timeout branch from a3d1693 to cae9e2b Compare October 25, 2024 21:35

brandond requested review from cwayne18, dereknola, manuelbuil and a team October 25, 2024 21:35

Bump etcd to v3.5.16-k3s1

518b88d

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

brandond force-pushed the etcd-defrag-no-timeout branch from cae9e2b to 521d515 Compare October 25, 2024 21:40

brandond mentioned this pull request Oct 25, 2024

[WIP] Bump K3s version for etcd-defrag-no-timeout rancher/rke2#7139

Closed

brandond force-pushed the etcd-defrag-no-timeout branch from 521d515 to 5e6dc72 Compare October 25, 2024 22:17

brandond force-pushed the etcd-defrag-no-timeout branch from 5e6dc72 to eede569 Compare October 26, 2024 00:28

dereknola previously approved these changes Oct 28, 2024

View reviewed changes

brandond dismissed dereknola’s stale review via 72eb27f October 28, 2024 19:51

brandond force-pushed the etcd-defrag-no-timeout branch from eede569 to 72eb27f Compare October 28, 2024 19:51

brandond requested a review from dereknola October 28, 2024 19:51

brandond added 2 commits October 28, 2024 21:58

Add tests for ETCD.Test()

09cea43

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

brandond force-pushed the etcd-defrag-no-timeout branch from 72eb27f to 09cea43 Compare October 28, 2024 21:58

manuelbuil approved these changes Oct 29, 2024

View reviewed changes

vitorsavian reviewed Oct 29, 2024

View reviewed changes

pkg/etcd/etcd.go Show resolved Hide resolved

vitorsavian approved these changes Oct 30, 2024

View reviewed changes

brandond merged commit a39e191 into k3s-io:master Oct 30, 2024
30 checks passed

This was referenced Nov 6, 2024

[release-1.31] Backports for 2024-11 #11261

Merged

[release-1.30] Backports for 2024-11 #11262

Merged

[release-1.29] Backports for 2024-11 #11263

Merged

[release-1.28] Backports for 2024-11 #11264

Merged

fmoral2 reviewed Nov 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix timeout when defragmenting etcd on startup #11164

Fix timeout when defragmenting etcd on startup #11164

brandond commented Oct 24, 2024 •

edited

Loading

brandond Oct 24, 2024 •

edited

Loading

codecov bot commented Oct 24, 2024 •

edited

Loading

manuelbuil commented Oct 24, 2024

brandond commented Oct 24, 2024 •

edited

Loading

brandond commented Oct 25, 2024 •

edited

Loading

fmoral2 Nov 11, 2024

brandond Nov 11, 2024

Fix timeout when defragmenting etcd on startup #11164

Fix timeout when defragmenting etcd on startup #11164

Conversation

brandond commented Oct 24, 2024 • edited Loading

Proposed Changes

Types of Changes

Verification

Testing

Linked Issues

User-Facing Change

Further Comments

brandond Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Oct 24, 2024 • edited Loading

Codecov Report

manuelbuil commented Oct 24, 2024

brandond commented Oct 24, 2024 • edited Loading

brandond commented Oct 25, 2024 • edited Loading

fmoral2 Nov 11, 2024

Choose a reason for hiding this comment

brandond Nov 11, 2024

Choose a reason for hiding this comment

brandond commented Oct 24, 2024 •

edited

Loading

brandond Oct 24, 2024 •

edited

Loading

codecov bot commented Oct 24, 2024 •

edited

Loading

brandond commented Oct 24, 2024 •

edited

Loading

brandond commented Oct 25, 2024 •

edited

Loading