Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Pod role not correct for quite a long time after stop leader node #3575

Closed
ahjing99 opened this issue Jun 5, 2023 · 0 comments · Fixed by #3971
Closed

[BUG]Pod role not correct for quite a long time after stop leader node #3575

ahjing99 opened this issue Jun 5, 2023 · 0 comments · Fixed by #3971
Assignees
Labels
bug kind/bug Something isn't working stability stability related issues
Milestone

Comments

@ahjing99
Copy link
Collaborator

ahjing99 commented Jun 5, 2023

➜ ~ kbcli version
Kubernetes: v1.25.8-gke.500
KubeBlocks: 0.6.0-alpha.10
kbcli: 0.6.0-alpha.10

  1. Before inject node stop ,yjtest-mysql-0 is the leader
  2. Stop the node where leader yjtest-mysql-0 reside
➜  ~ kbcli fault node stop gke-yjtest-default-pool-eaab2fad-bxg9 -c=gcp --region=us-central1-c --project=apecloud-platform-engineering
Secret cloud-key-secret-gcp exists under default namespace.
GCPChaos node-chaos-6rxhc created
  1. node stopped
➜  ~ kbcli cluster connect yjtest
Connect to instance yjtest-mysql-1: out of yjtest-mysql-1(leader), yjtest-mysql-2(follower)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 75
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 3e93ae7
  1. node started,but the role of the original leader pod is not correct
➜  ~ kbcli cluster describe yjtest
Name: yjtest	 Created Time: Jun 05,2023 15:49 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   WipeOut

Endpoints:
COMPONENT   MODE        INTERNAL                                      EXTERNAL
mysql       ReadWrite   [yjtest-mysql.default.svc.cluster.local:3306](http://yjtest-mysql.default.svc.cluster.local:3306/)   <none>

Topology:
COMPONENT   INSTANCE         ROLE       STATUS    AZ              NODE                                                  CREATED-TIME
mysql       yjtest-mysql-0   leader     Running   us-central1-c   gke-yjtest-default-pool-eaab2fad-bxg9/[10.128.0.15](http://10.128.0.15/)     Jun 05,2023 15:49 UTC+0800
mysql       yjtest-mysql-1   leader     Running   us-central1-c   gke-yjtest-default-pool-eaab2fad-2618/[10.128.0.2](http://10.128.0.2/)      Jun 05,2023 15:49 UTC+0800
mysql       yjtest-mysql-2   follower   Running   us-central1-c   gke-yjtest-default-pool-eaab2fad-4rxv/[10.128.15.229](http://10.128.15.229/)   Jun 05,2023 15:49 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       2 / 2                8Gi / 8Gi               data:5Gi       standard-rwo

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   [registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8](http://registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8)

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default yjtest


➜  ~ kbcli cluster connect yjtest
Connect to instance yjtest-mysql-1: out of yjtest-mysql-1(leader), yjtest-mysql-2(follower)
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 192
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 3e93ae7

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from information_schema.wesql_cluster_global;\
+-----------+--------------------------------------------+-------------+------------+----------+-----------+------------+-----------------+----------------+---------------+------------+--------------+
| SERVER_ID | IP_PORT                                    | MATCH_INDEX | NEXT_INDEX | ROLE     | HAS_VOTED | FORCE_SYNC | ELECTION_WEIGHT | LEARNER_SOURCE | APPLIED_INDEX | PIPELINING | SEND_APPLIED |
+-----------+--------------------------------------------+-------------+------------+----------+-----------+------------+-----------------+----------------+---------------+------------+--------------+
|         1 | yjtest-mysql-0.yjtest-mysql-headless:13306 |          17 |         18 | Follower | No        | No         |               5 |              0 |            16 | Yes        | No           |
|         2 | yjtest-mysql-1.yjtest-mysql-headless:13306 |          17 |          0 | Leader   | Yes       | No         |               5 |              0 |            14 | No         | No           |
|         3 | yjtest-mysql-2.yjtest-mysql-headless:13306 |          17 |         18 | Follower | Yes       | No         |               5 |              0 |            15 | Yes        | No           |
+-----------+--------------------------------------------+-------------+------------+----------+-----------+------------+-----------------+----------------+---------------+------------+--------------+
3 rows in set (0.00 sec)


➜  ~ k describe pod yjtest-mysql-0
Name:         yjtest-mysql-0
Namespace:    default
Priority:     0
Node:         gke-yjtest-default-pool-eaab2fad-bxg9/10.128.0.15
Start Time:   Mon, 05 Jun 2023 15:49:22 +0800
Labels:       app.kubernetes.io/component=mysql
              app.kubernetes.io/instance=yjtest
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=apecloud-mysql
              app.kubernetes.io/version=ac-mysql-8.0.30
              apps.kubeblocks.io/component-name=mysql
              apps.kubeblocks.io/workload-type=Consensus
              controller-revision-hash=yjtest-mysql-7f5dfc5c59
              cs.apps.kubeblocks.io/access-mode=ReadWrite
              kubeblocks.io/role=leader
              statefulset.kubernetes.io/pod-name=yjtest-mysql-0
Annotations:  apps.kubeblocks.io/component-replicas: 3
              cs.apps.kubeblocks.io/leader: yjtest-mysql-1
Status:       Running
IP:           10.104.2.5
IPs:
  IP:           10.104.2.5
Controlled By:  StatefulSet/yjtest-mysql
Containers:
  mysql:
    Container ID:  containerd://910f94f0c0cb0e4bd7829482bd97a320e07472210dd1bb8a4521c72e70dece82
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server@sha256:5f68384d13b931956ff7e1726f90e71dd956a582136db6cf824ad48ef5e0ec74
    Ports:         3306/TCP, 13306/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/setup.sh
    State:          Running
      Started:      Mon, 05 Jun 2023 16:13:45 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:     2
      memory:  8Gi
    Environment Variables from:
      yjtest-mysql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               yjtest-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           yjtest
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      yjtest-mysql
      KB_CLUSTER_UID_POSTFIX_8:  59b2d2f2
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MYSQL_ROOT_HOST:           %
      MYSQL_ROOT_USER:           <set to the key 'username' in secret 'yjtest-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:       <set to the key 'password' in secret 'yjtest-conn-credential'>  Optional: false
      MYSQL_DATABASE:            mydb
      MYSQL_USER:                u1
      MYSQL_PASSWORD:            u1
      CLUSTER_ID:                1
      CLUSTER_START_INDEX:       1
      REPLICATION_USER:          replicator
      REPLICATION_PASSWORD:
      MYSQL_TEMPLATE_CONFIG:
      MYSQL_CUSTOM_CONFIG:
      MYSQL_DYNAMIC_CONFIG:
      KB_EMBEDDED_WESQL:         1
    Mounts:
      /data/mysql from data (rw)
      /etc/annotations from annotations (rw)
      /opt/mysql from mysql-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
  metrics:
    Container ID:  containerd://6f97a649c695756dbea22edb51ec7e106d3eb1222eb3e95f915a0b4b393e0b3f
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter@sha256:8a394d7e722a3c6be7082485eafb4c4082af6f2caf49b30446472ffeedc2ea28
    Port:          9104/TCP
    Host Port:     0/TCP
    Command:
      /bin/mysqld_exporter
      --collect.info_schema.wesql_consensus
    State:          Running
      Started:      Mon, 05 Jun 2023 16:13:59 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6
    Readiness:      http-get http://:http-metrics/ delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment Variables from:
      yjtest-mysql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               yjtest-mysql-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           yjtest
      KB_COMP_NAME:              mysql
      KB_CLUSTER_COMP_NAME:      yjtest-mysql
      KB_CLUSTER_UID_POSTFIX_8:  59b2d2f2
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MYSQL_USER:                <set to the key 'username' in secret 'yjtest-conn-credential'>  Optional: false
      MYSQL_PASSWORD:            <set to the key 'password' in secret 'yjtest-conn-credential'>  Optional: false
      DATA_SOURCE_NAME:          $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
  kb-checkrole:
    Container ID:  containerd://6e56b1c9d631f81941159b04708d448b950cde11a7588aaa27fc48f33aa31179
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:b9d86f3c8ae72a8609ba50ec65d6b71e84734e25d81f160383aef1eba693b6e4
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      probe
      --app-id
      batch-sdk
      --dapr-http-port
      3501
      --dapr-grpc-port
      50001
      --log-level
      info
      --config
      /config/probe/config.yaml
      --components-path
      /config/probe/components
    State:          Running
      Started:      Mon, 05 Jun 2023 16:14:27 +0800
    Ready:          True
    Restart Count:  0
    Readiness:      exec [curl -X POST --max-time 1 --fail-with-body --silent -H Content-ComponentDefRef: application/json http://localhost:3501/v1.0/bindings/mysql -d {"operation": "checkRole", "metadata":{"sql":""}}] delay=0s timeout=1s period=1s #success=1 #failure=2
    Startup:        tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      yjtest-mysql-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                yjtest-mysql-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            yjtest
      KB_COMP_NAME:               mysql
      KB_CLUSTER_COMP_NAME:       yjtest-mysql
      KB_CLUSTER_UID_POSTFIX_8:   59b2d2f2
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'yjtest-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'yjtest-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            3306
      KB_SERVICE_ROLES:           {"follower":"Readonly","leader":"ReadWrite","learner":"Readonly"}
      KB_SERVICE_CHARACTER_TYPE:  mysql
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
  config-manager:
    Container ID:  containerd://30dabd11e3b933123bd2bbd077b8806c8413281a487c32adc8ebf7bc64632ebf
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:b9d86f3c8ae72a8609ba50ec65d6b71e84734e25d81f160383aef1eba693b6e4
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/reloader
    Args:
      --operator-update-enable
      --log-level
      info
      --tcp
      9901
      --notify-type
      tpl
      --tpl-config
      /opt/config/reload/reload.yaml
    State:          Running
      Started:      Mon, 05 Jun 2023 16:14:27 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      CONFIG_MANAGER_POD_IP:   (v1:status.podIP)
      DB_TYPE:                mysql
      MYSQL_USER:             <set to the key 'username' in secret 'yjtest-conn-credential'>  Optional: false
      MYSQL_PASSWORD:         <set to the key 'password' in secret 'yjtest-conn-credential'>  Optional: false
      DATA_SOURCE_NAME:       $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
    Mounts:
      /opt/config/reload from reload-manager-reload (rw)
      /opt/mysql from mysql-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z79cn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-yjtest-mysql-0
    ReadOnly:   false
  annotations:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      yjtest-mysql-apecloud-mysql-scripts
    Optional:  false
  mysql-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      yjtest-mysql-mysql-consensusset-config
    Optional:  false
  reload-manager-reload:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-reload-script-yjtest
    Optional:  false
  kube-api-access-z79cn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Normal   Scheduled               51m                default-scheduler        Successfully assigned default/yjtest-mysql-0 to gke-yjtest-default-pool-eaab2fad-bxg9
  Normal   SuccessfulAttachVolume  51m                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-caa4a7d5-37d1-44ab-844b-b46e8bc15243"
  Normal   Started                 51m                kubelet                  Started container metrics
  Normal   Created                 51m                kubelet                  Created container mysql
  Normal   Started                 51m                kubelet                  Started container mysql
  Normal   Pulled                  51m                kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1" already present on machine
  Normal   Created                 51m                kubelet                  Created container metrics
  Normal   Pulled                  51m                kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8" already present on machine
  Normal   Pulled                  51m                kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" already present on machine
  Normal   Created                 51m                kubelet                  Created container kb-checkrole
  Normal   Started                 51m                kubelet                  Started container kb-checkrole
  Normal   Pulled                  51m                kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" already present on machine
  Normal   Created                 51m                kubelet                  Created container config-manager
  Normal   Started                 51m                kubelet                  Started container config-manager
  Normal   checkRole               51m                sqlchannel               {"event":"Success","operation":"checkRole","originalRole":"","role":"Follower"}
  Normal   checkRole               49m                sqlchannel               {"event":"Success","operation":"checkRole","originalRole":"Follower","role":"Leader"}
  Warning  NodeNotReady            30m                node-controller          Node is not ready
  Warning  NetworkNotReady         28m                kubelet                  network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  Warning  FailedMount             28m                kubelet                  MountVolume.SetUp failed for volume "mysql-config" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount             28m                kubelet                  MountVolume.SetUp failed for volume "reload-manager-reload" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount             27m (x6 over 28m)  kubelet                  MountVolume.NodeAffinity check failed for volume "pvc-caa4a7d5-37d1-44ab-844b-b46e8bc15243" : no matching NodeSelectorTerms
  Normal   Pulling                 27m                kubelet                  Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8"
  Normal   Started                 27m                kubelet                  Started container mysql
  Normal   Pulling                 27m                kubelet                  Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1"
  Normal   Pulled                  27m                kubelet                  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha8.20230523.g3e93ae7.8" in 30.682453577s (30.682461217s including waiting)
  Normal   Created                 27m                kubelet                  Created container mysql
  Normal   Pulled                  26m                kubelet                  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/mysqld-exporter:0.14.1" in 14.588519462s (14.588534531s including waiting)
  Normal   Created                 26m                kubelet                  Created container metrics
  Normal   Pulling                 26m                kubelet                  Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10"
  Normal   Started                 26m                kubelet                  Started container metrics
  Normal   Started                 26m                kubelet                  Started container kb-checkrole
  Normal   Pulled                  26m                kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" already present on machine
  Normal   Created                 26m                kubelet                  Created container config-manager
  Normal   Started                 26m                kubelet                  Started container config-manager
  Normal   Created                 26m                kubelet                  Created container kb-checkrole
  Normal   Pulled                  26m                kubelet                  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.10" in 27.195377504s (27.195396896s including waiting)

➜ ~ k logs yjtest-mysql-0 -c kb-checkrole >kb.log
kb.log

@ahjing99 ahjing99 added the kind/bug Something isn't working label Jun 5, 2023
@ahjing99 ahjing99 added this to the Release 0.6.0 milestone Jun 5, 2023
@lynnleelhl lynnleelhl assigned free6om and unassigned lynnleelhl Jun 5, 2023
@xuriwuyun xuriwuyun linked a pull request Jun 27, 2023 that will close this issue
@ahjing99 ahjing99 added the stability stability related issues label Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug kind/bug Something isn't working stability stability related issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants