You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
➜ ~ kbcli version
Kubernetes: v1.25.8-gke.500
KubeBlocks: 0.6.0-alpha.10
kbcli: 0.6.0-alpha.10
Before inject node stop ,yjtest-mysql-0 is the leader
Stop the node where leader yjtest-mysql-0 reside
➜ ~ kbcli fault node stop gke-yjtest-default-pool-eaab2fad-bxg9 -c=gcp --region=us-central1-c --project=apecloud-platform-engineering
Secret cloud-key-secret-gcp exists under default namespace.
GCPChaos node-chaos-6rxhc created
node stopped
➜ ~ kbcli cluster connect yjtest
Connect to instance yjtest-mysql-1: out of yjtest-mysql-1(leader), yjtest-mysql-2(follower)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 75
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 3e93ae7
node started,but the role of the original leader pod is not correct
➜ ~ kbcli cluster describe yjtest
Name: yjtest Created Time: Jun 05,2023 15:49 UTC+0800
NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default apecloud-mysql ac-mysql-8.0.30 Running WipeOut
Endpoints:
COMPONENT MODE INTERNAL EXTERNAL
mysql ReadWrite [yjtest-mysql.default.svc.cluster.local:3306](http://yjtest-mysql.default.svc.cluster.local:3306/) <none>
Topology:
COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
mysql yjtest-mysql-0 leader Running us-central1-c gke-yjtest-default-pool-eaab2fad-bxg9/[10.128.0.15](http://10.128.0.15/) Jun 05,2023 15:49 UTC+0800
mysql yjtest-mysql-1 leader Running us-central1-c gke-yjtest-default-pool-eaab2fad-2618/[10.128.0.2](http://10.128.0.2/) Jun 05,2023 15:49 UTC+0800
mysql yjtest-mysql-2 follower Running us-central1-c gke-yjtest-default-pool-eaab2fad-4rxv/[10.128.15.229](http://10.128.15.229/) Jun 05,2023 15:49 UTC+0800
Resources Allocation:
COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
mysql false 2 / 2 8Gi / 8Gi data:5Gi standard-rwo
Images:
COMPONENT TYPE IMAGE
mysql mysql [registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8](http://registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8)
Data Protection:
AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME
Disabled <none> <none> 7d <none> <none>
Show cluster events: kbcli cluster list-events -n default yjtest
➜ ~ kbcli cluster connect yjtest
Connect to instance yjtest-mysql-1: out of yjtest-mysql-1(leader), yjtest-mysql-2(follower)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 192
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 3e93ae7
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select * from information_schema.wesql_cluster_global;\
+-----------+--------------------------------------------+-------------+------------+----------+-----------+------------+-----------------+----------------+---------------+------------+--------------+
| SERVER_ID | IP_PORT | MATCH_INDEX | NEXT_INDEX | ROLE | HAS_VOTED | FORCE_SYNC | ELECTION_WEIGHT | LEARNER_SOURCE | APPLIED_INDEX | PIPELINING | SEND_APPLIED |
+-----------+--------------------------------------------+-------------+------------+----------+-----------+------------+-----------------+----------------+---------------+------------+--------------+
| 1 | yjtest-mysql-0.yjtest-mysql-headless:13306 | 17 | 18 | Follower | No | No | 5 | 0 | 16 | Yes | No |
| 2 | yjtest-mysql-1.yjtest-mysql-headless:13306 | 17 | 0 | Leader | Yes | No | 5 | 0 | 14 | No | No |
| 3 | yjtest-mysql-2.yjtest-mysql-headless:13306 | 17 | 18 | Follower | Yes | No | 5 | 0 | 15 | Yes | No |
+-----------+--------------------------------------------+-------------+------------+----------+-----------+------------+-----------------+----------------+---------------+------------+--------------+
3 rows in set (0.00 sec)
➜ ~ k describe pod yjtest-mysql-0
Name: yjtest-mysql-0
Namespace: default
Priority: 0
Node: gke-yjtest-default-pool-eaab2fad-bxg9/10.128.0.15
Start Time: Mon, 05 Jun 2023 15:49:22 +0800
Labels: app.kubernetes.io/component=mysql
app.kubernetes.io/instance=yjtest
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=apecloud-mysql
app.kubernetes.io/version=ac-mysql-8.0.30
apps.kubeblocks.io/component-name=mysql
apps.kubeblocks.io/workload-type=Consensus
controller-revision-hash=yjtest-mysql-7f5dfc5c59
cs.apps.kubeblocks.io/access-mode=ReadWrite
kubeblocks.io/role=leader
statefulset.kubernetes.io/pod-name=yjtest-mysql-0
Annotations: apps.kubeblocks.io/component-replicas: 3
cs.apps.kubeblocks.io/leader: yjtest-mysql-1
Status: Running
IP: 10.104.2.5
IPs:
IP: 10.104.2.5
Controlled By: StatefulSet/yjtest-mysql
Containers:
mysql:
Container ID: containerd://910f94f0c0cb0e4bd7829482bd97a320e07472210dd1bb8a4521c72e70dece82
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server@sha256:5f68384d13b931956ff7e1726f90e71dd956a582136db6cf824ad48ef5e0ec74
Ports: 3306/TCP, 13306/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/scripts/setup.sh
State: Running
Started: Mon, 05 Jun 2023 16:13:45 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 8Gi
Requests:
cpu: 2
memory: 8Gi
Environment Variables from:
yjtest-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: yjtest-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: yjtest
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: yjtest-mysql
KB_CLUSTER_UID_POSTFIX_8: 59b2d2f2
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
MYSQL_ROOT_HOST: %
MYSQL_ROOT_USER: <set to the key 'username' in secret 'yjtest-conn-credential'> Optional: false
MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'yjtest-conn-credential'> Optional: false
MYSQL_DATABASE: mydb
MYSQL_USER: u1
MYSQL_PASSWORD: u1
CLUSTER_ID: 1
CLUSTER_START_INDEX: 1
REPLICATION_USER: replicator
REPLICATION_PASSWORD:
MYSQL_TEMPLATE_CONFIG:
MYSQL_CUSTOM_CONFIG:
MYSQL_DYNAMIC_CONFIG:
KB_EMBEDDED_WESQL: 1
Mounts:
/data/mysql from data (rw)
/etc/annotations from annotations (rw)
/opt/mysql from mysql-config (rw)
/scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
metrics:
Container ID: containerd://6f97a649c695756dbea22edb51ec7e106d3eb1222eb3e95f915a0b4b393e0b3f
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter@sha256:8a394d7e722a3c6be7082485eafb4c4082af6f2caf49b30446472ffeedc2ea28
Port: 9104/TCP
Host Port: 0/TCP
Command:
/bin/mysqld_exporter
--collect.info_schema.wesql_consensus
State: Running
Started: Mon, 05 Jun 2023 16:13:59 +0800
Ready: True
Restart Count: 0
Liveness: http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6
Readiness: http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6
Environment Variables from:
yjtest-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: yjtest-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: yjtest
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: yjtest-mysql
KB_CLUSTER_UID_POSTFIX_8: 59b2d2f2
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
MYSQL_USER: <set to the key 'username' in secret 'yjtest-conn-credential'> Optional: false
MYSQL_PASSWORD: <set to the key 'password' in secret 'yjtest-conn-credential'> Optional: false
DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
kb-checkrole:
Container ID: containerd://6e56b1c9d631f81941159b04708d448b950cde11a7588aaa27fc48f33aa31179
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:b9d86f3c8ae72a8609ba50ec65d6b71e84734e25d81f160383aef1eba693b6e4
Ports: 3501/TCP, 50001/TCP
Host Ports: 0/TCP, 0/TCP
Command:
probe
--app-id
batch-sdk
--dapr-http-port
3501
--dapr-grpc-port
50001
--log-level
info
--config
/config/probe/config.yaml
--components-path
/config/probe/components
State: Running
Started: Mon, 05 Jun 2023 16:14:27 +0800
Ready: True
Restart Count: 0
Readiness: exec [curl -X POST --max-time 1 --fail-with-body --silent -H Content-ComponentDefRef: application/json http://localhost:3501/v1.0/bindings/mysql -d {"operation": "checkRole", "metadata":{"sql":""}}] delay=0s timeout=1s period=1s #success=1 #failure=2
Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
yjtest-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: yjtest-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: yjtest
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: yjtest-mysql
KB_CLUSTER_UID_POSTFIX_8: 59b2d2f2
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
KB_SERVICE_USER: <set to the key 'username' in secret 'yjtest-conn-credential'> Optional: false
KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'yjtest-conn-credential'> Optional: false
KB_SERVICE_PORT: 3306
KB_SERVICE_ROLES: {"follower":"Readonly","leader":"ReadWrite","learner":"Readonly"}
KB_SERVICE_CHARACTER_TYPE: mysql
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
config-manager:
Container ID: containerd://30dabd11e3b933123bd2bbd077b8806c8413281a487c32adc8ebf7bc64632ebf
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10
Image ID: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:b9d86f3c8ae72a8609ba50ec65d6b71e84734e25d81f160383aef1eba693b6e4
Port: <none>
Host Port: <none>
Command:
/bin/reloader
Args:
--operator-update-enable
--log-level
info
--tcp
9901
--notify-type
tpl
--tpl-config
/opt/config/reload/reload.yaml
State: Running
Started: Mon, 05 Jun 2023 16:14:27 +0800
Ready: True
Restart Count: 0
Environment:
CONFIG_MANAGER_POD_IP: (v1:status.podIP)
DB_TYPE: mysql
MYSQL_USER: <set to the key 'username' in secret 'yjtest-conn-credential'> Optional: false
MYSQL_PASSWORD: <set to the key 'password' in secret 'yjtest-conn-credential'> Optional: false
DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
Mounts:
/opt/config/reload from reload-manager-reload (rw)
/opt/mysql from mysql-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-yjtest-mysql-0
ReadOnly: false
annotations:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader
metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: yjtest-mysql-apecloud-mysql-scripts
Optional: false
mysql-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: yjtest-mysql-mysql-consensusset-config
Optional: false
reload-manager-reload:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mysql-reload-script-yjtest
Optional: false
kube-api-access-z79cn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 51m default-scheduler Successfully assigned default/yjtest-mysql-0 to gke-yjtest-default-pool-eaab2fad-bxg9
Normal SuccessfulAttachVolume 51m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-caa4a7d5-37d1-44ab-844b-b46e8bc15243"
Normal Started 51m kubelet Started container metrics
Normal Created 51m kubelet Created container mysql
Normal Started 51m kubelet Started container mysql
Normal Pulled 51m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1" already present on machine
Normal Created 51m kubelet Created container metrics
Normal Pulled 51m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8" already present on machine
Normal Pulled 51m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" already present on machine
Normal Created 51m kubelet Created container kb-checkrole
Normal Started 51m kubelet Started container kb-checkrole
Normal Pulled 51m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" already present on machine
Normal Created 51m kubelet Created container config-manager
Normal Started 51m kubelet Started container config-manager
Normal checkRole 51m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"","role":"Follower"}
Normal checkRole 49m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"Follower","role":"Leader"}
Warning NodeNotReady 30m node-controller Node is not ready
Warning NetworkNotReady 28m kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Warning FailedMount 28m kubelet MountVolume.SetUp failed for volume "mysql-config" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 28m kubelet MountVolume.SetUp failed for volume "reload-manager-reload" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 27m (x6 over 28m) kubelet MountVolume.NodeAffinity check failed for volume "pvc-caa4a7d5-37d1-44ab-844b-b46e8bc15243" : no matching NodeSelectorTerms
Normal Pulling 27m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8"
Normal Started 27m kubelet Started container mysql
Normal Pulling 27m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1"
Normal Pulled 27m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8" in 30.682453577s (30.682461217s including waiting)
Normal Created 27m kubelet Created container mysql
Normal Pulled 26m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1" in 14.588519462s (14.588534531s including waiting)
Normal Created 26m kubelet Created container metrics
Normal Pulling 26m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10"
Normal Started 26m kubelet Started container metrics
Normal Started 26m kubelet Started container kb-checkrole
Normal Pulled 26m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" already present on machine
Normal Created 26m kubelet Created container config-manager
Normal Started 26m kubelet Started container config-manager
Normal Created 26m kubelet Created container kb-checkrole
Normal Pulled 26m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" in 27.195377504s (27.195396896s including waiting)
➜ ~ k logs yjtest-mysql-0 -c kb-checkrole >kb.log kb.log
The text was updated successfully, but these errors were encountered:
➜ ~ kbcli version
Kubernetes: v1.25.8-gke.500
KubeBlocks: 0.6.0-alpha.10
kbcli: 0.6.0-alpha.10
➜ ~ k logs yjtest-mysql-0 -c kb-checkrole >kb.log
kb.log
The text was updated successfully, but these errors were encountered: