Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]switchover fail when scaleIn #8116

Open
loomts opened this issue Sep 10, 2024 · 0 comments
Open

[BUG]switchover fail when scaleIn #8116

loomts opened this issue Sep 10, 2024 · 0 comments
Assignees
@loomts @Ursasi
Labels
kind/bug Something isn't working

Comments

@loomts
Copy link
Contributor

loomts commented Sep 10, 2024

Describe the bug
when etcd scaleIn trigger switchover, will run switchover command in kbagent, but not success

To Reproduce
Steps to reproduce the behavior:

  1. setup kb environment
kb version
Kubernetes: v1.30.0
KubeBlocks: 1.0.0-alpha.4
kbcli: 1.0.0-alpha.0
  1. use latest main branch of kb
  2. make build-tools-image, build-manager-image and change the tag, then cache them in minikube, let it become runtime images of kubeblocks-tools and kubeblocks
  3. use this branch of addons chore: etcd support newapi kubeblocks-addons#941
  4. deploy etcd helm upgrade -i etcd addons/etcd && helm upgrade -i etcd-cluster addons-cluster/etcd and apply hscale kubectl apply -f examples/etcd/horizontalscale.yaml

Expected behavior
after apply horizontalscale.yaml, scaleIn will make it switchover and shutdown

Screenshots
restart random pod several times, but do nothing
image

Additional context

k logs kubeblocks-64bc7dbb75-vpx4d -n kb-system
2024-09-09T16:38:53.785Z	INFO	leave member at scaling-in error, retry later: switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "57e1fe67-303b-4930-9171-bd30cb3688ce", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:53.790Z	INFO	switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "57e1fe67-303b-4930-9171-bd30cb3688ce", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:53.790Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "57e1fe67-303b-4930-9171-bd30cb3688ce", "error": "switchover succeed, wait role label to be updated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:53.905Z	INFO	leave member at scaling-in error, retry later: switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "aef31d24-f235-4845-9ec1-60d6aa2d1021", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:53.911Z	INFO	switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "aef31d24-f235-4845-9ec1-60d6aa2d1021", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:53.911Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "aef31d24-f235-4845-9ec1-60d6aa2d1021", "error": "switchover succeed, wait role label to be updated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:53.994Z	INFO	leave member at scaling-in error, retry later: switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "68c99898-309f-4c04-b0a6-d0852390e79f", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:53.998Z	INFO	switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "68c99898-309f-4c04-b0a6-d0852390e79f", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:53.998Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "68c99898-309f-4c04-b0a6-d0852390e79f", "error": "switchover succeed, wait role label to be updated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:54.087Z	INFO	leave member at scaling-in error, retry later: switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "189ac048-4379-4f5a-9580-3f11919f602f", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:54.091Z	INFO	switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "189ac048-4379-4f5a-9580-3f11919f602f", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:54.091Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "189ac048-4379-4f5a-9580-3f11919f602f", "error": "switchover succeed, wait role label to be updated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:54.259Z	INFO	leave member at scaling-in error, retry later: switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "f6fa7b6d-407b-4967-b336-a7423eb94e8d", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:54.264Z	INFO	switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "f6fa7b6d-407b-4967-b336-a7423eb94e8d", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:54.264Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "f6fa7b6d-407b-4967-b336-a7423eb94e8d", "error": "switchover succeed, wait role label to be updated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:54.980Z	INFO	leave member at scaling-in error, retry later: switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "77da5687-7220-46ef-b65f-e83a377a7aa9", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:54.984Z	INFO	switchover succeed, wait role label to be updated	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "77da5687-7220-46ef-b65f-e83a377a7aa9", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:54.984Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "77da5687-7220-46ef-b65f-e83a377a7aa9", "error": "switchover succeed, wait role label to be updated"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:55.147Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "f76f8fa6-4c8c-4652-8339-745b9aced69e", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:55.151Z	INFO	http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "f76f8fa6-4c8c-4652-8339-745b9aced69e", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:55.151Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "f76f8fa6-4c8c-4652-8339-745b9aced69e", "error": "http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).Switchover\n\t/src/pkg/controller/component/lifecycle/kbagent.go:88\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn.func2\n\t/src/controllers/apps/transformer_component_workload.go:610\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:644\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:38:56.285Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "22d07e5f-4273-4f11-941b-4d386a0c9ab6", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:56.289Z	INFO	http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "22d07e5f-4273-4f11-941b-4d386a0c9ab6", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:38:56.289Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "22d07e5f-4273-4f11-941b-4d386a0c9ab6", "error": "http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).Switchover\n\t/src/pkg/controller/component/lifecycle/kbagent.go:88\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn.func2\n\t/src/controllers/apps/transformer_component_workload.go:610\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:644\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:01.432Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "ca36bec1-5302-4d17-ab2e-f6da71ad1d3a", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:01.435Z	INFO	http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "ca36bec1-5302-4d17-ab2e-f6da71ad1d3a", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:01.435Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "ca36bec1-5302-4d17-ab2e-f6da71ad1d3a", "error": "http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).Switchover\n\t/src/pkg/controller/component/lifecycle/kbagent.go:88\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn.func2\n\t/src/controllers/apps/transformer_component_workload.go:610\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:644\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:11.696Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "8d36f703-1f64-477f-9036-0743f69055a8", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:11.699Z	INFO	http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "8d36f703-1f64-477f-9036-0743f69055a8", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:11.699Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "8d36f703-1f64-477f-9036-0743f69055a8", "error": "http error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action switchover at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).Switchover\n\t/src/pkg/controller/component/lifecycle/kbagent.go:88\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn.func2\n\t/src/controllers/apps/transformer_component_workload.go:610\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:644\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:11.841Z	INFO	handle role change event	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-cluster-etcd-etcd-0-0.vrx8gz9r2b8vdllw","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd-etcd-0-0.vrx8gz9r2b8vdllw", "reconcileID": "640f95b9-aa2b-41f1-b115-931de2ea83fc", "event": {"name":"etcd-cluster-etcd-etcd-0-0.vrx8gz9r2b8vdllw","namespace":"default"}, "pod": "etcd-cluster-etcd-etcd-0-0", "role": "follower", "originalRole": ""}
2024-09-09T16:39:11.885Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "namespace": "default", "name": "etcd-horizontalscaling", "reconcileID": "a47cfb4b-657c-4f94-a20d-e774199143f6", "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}}
2024-09-09T16:39:11.914Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "0abc70ba-ca60-4bb4-9497-041a2b41be8b", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:11.921Z	INFO	http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "0abc70ba-ca60-4bb4-9497-041a2b41be8b", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:11.921Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "0abc70ba-ca60-4bb4-9497-041a2b41be8b", "error": "http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).MemberLeave\n\t/src/pkg/controller/component/lifecycle/kbagent.go:108\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:648\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:11.944Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "2ff8e171-c734-407e-9bd6-3367c5074449", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:11.948Z	INFO	http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "2ff8e171-c734-407e-9bd6-3367c5074449", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:11.948Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "2ff8e171-c734-407e-9bd6-3367c5074449", "error": "http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).MemberLeave\n\t/src/pkg/controller/component/lifecycle/kbagent.go:108\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:648\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:23.651Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "namespace": "default", "name": "etcd-horizontalscaling", "reconcileID": "46ebd4ab-bb9e-4d54-9846-8124055b4c40", "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}}
2024-09-09T16:39:23.790Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "6354b91d-04fd-48b3-a91b-14d8d1840daa", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:23.796Z	INFO	http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "6354b91d-04fd-48b3-a91b-14d8d1840daa", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:23.796Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "6354b91d-04fd-48b3-a91b-14d8d1840daa", "error": "http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).MemberLeave\n\t/src/pkg/controller/component/lifecycle/kbagent.go:108\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:648\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:23.799Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "namespace": "default", "name": "etcd-horizontalscaling", "reconcileID": "05391cd5-5094-46f6-a88f-ae0cb1aa96ae", "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}}
2024-09-09T16:39:23.826Z	INFO	leave member at scaling-in error, retry later: http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "c558d457-bd45-41d3-bc9c-8d7af78b69db", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:23.831Z	INFO	http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "c558d457-bd45-41d3-bc9c-8d7af78b69db", "component": {"name":"etcd-cluster-etcd","namespace":"default"}}
2024-09-09T16:39:23.831Z	ERROR	Reconciler error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"etcd-cluster-etcd","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd", "reconcileID": "c558d457-bd45-41d3-bc9c-8d7af78b69db", "error": "http error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0: unexpected http status code: 503 Service Unavailable", "errorVerbose": "unexpected http status code: 503 Service Unavailable\nhttp error occurred when executing action memberLeave at pod etcd-cluster-etcd-etcd-0-0\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/component/lifecycle/kbagent.go:298\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:215\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/component/lifecycle/kbagent.go:142\ngithub.com/apecloud/kubeblocks/pkg/controller/component/lifecycle.(*kbagent).MemberLeave\n\t/src/pkg/controller/component/lifecycle/kbagent.go:108\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).leaveMember4ScaleIn\n\t/src/controllers/apps/transformer_component_workload.go:648\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).scaleIn\n\t/src/controllers/apps/transformer_component_workload.go:526\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadOps).horizontalScale\n\t/src/controllers/apps/transformer_component_workload.go:447\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleWorkloadUpdate\n\t/src/controllers/apps/transformer_component_workload.go:204\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).handleUpdate\n\t/src/controllers/apps/transformer_component_workload.go:178\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentWorkloadTransformer).Transform\n\t/src/controllers/apps/transformer_component_workload.go:122\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component_plan_builder.go:117\ngithub.com/apecloud/kubeblocks/controllers/apps.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component_controller.go:193\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1197"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
2024-09-09T16:39:24.982Z	INFO	handle role change event	{"controller": "event", "controllerGroup": "", "controllerKind": "Event", "Event": {"name":"etcd-cluster-etcd-etcd-1-0.zfbzbs9jz9m7s5c8","namespace":"default"}, "namespace": "default", "name": "etcd-cluster-etcd-etcd-1-0.zfbzbs9jz9m7s5c8", "reconcileID": "fc10aa07-dcb6-4f09-8b1f-4c35d769a242", "event": {"name":"etcd-cluster-etcd-etcd-1-0.zfbzbs9jz9m7s5c8","namespace":"default"}, "pod": "etcd-cluster-etcd-etcd-1-0", "role": "", "originalRole": ""}
2024-09-09T16:39:25.029Z	INFO	reconcile	{"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "namespace": "default", "name": "etcd-horizontalscaling", "reconcileID": "7ae0a070-0a1b-4042-9068-60e3b4093b99", "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}, "opsRequest": {"name":"etcd-horizontalscaling","namespace":"default"}}
@loomts loomts added the kind/bug Something isn't working label Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@loomts loomts

@Ursasi Ursasi

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@loomts @Ursasi