Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]MongoDB vscale ops failed for pod crash #3949

Closed
ahjing99 opened this issue Jun 26, 2023 · 0 comments · Fixed by #3954 or #3956
Closed

[BUG]MongoDB vscale ops failed for pod crash #3949

ahjing99 opened this issue Jun 26, 2023 · 0 comments · Fixed by #3954 or #3956
Assignees
@xuriwuyun
Labels
bug kind/bug Something isn't working
Milestone
Release 0.6.0

Comments

@ahjing99
Copy link
Collaborator

kbcli version
Kubernetes: v1.25.8-gke.1000
KubeBlocks: 0.5.3-beta.7
kbcli: 0.5.3-beta.7

      `kbcli cluster create cluster-icegqr              --termination-policy=DoNotTerminate              --monitor=false --enable-all-logs=false --cluster-definition=mongodb --set type=mongodb,cpu=100m,memory=0.5Gi,replicas=3,storage=1Gi  --namespace default `

Info: --cluster-version is not specified, ClusterVersion mongodb-5.0.14 is applied by default
Cluster cluster-icegqr created

      `echo cluster-icegqr | kbcli cluster vscale cluster-icegqr                 --components mongodb                 --cpu 200m                 --memory 0.6Gi --namespace default `

Please type the name again(separate with white space when more than one): cluster-icegqr
OpsRequest cluster-icegqr-verticalscaling-q7xxp created successfully, you can view the progress:
	kbcli cluster describe-ops cluster-icegqr-verticalscaling-q7xxp -n default


k get ops
NAME                                   TYPE              CLUSTER          STATUS   PROGRESS   AGE
cluster-icegqr-verticalscaling-q7xxp   VerticalScaling   cluster-icegqr   Failed   3/3        6m48s

k describe ops cluster-icegqr-verticalscaling-q7xxp
Name:         cluster-icegqr-verticalscaling-q7xxp
Namespace:    default
Labels:       app.kubernetes.io/instance=cluster-icegqr
Annotations:  kubeblocks.io/reconcile: 2023-06-26T05:52:19.897434177Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2023-06-26T05:47:45Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  cluster-icegqr-verticalscaling-
  Generation:     2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
      f:spec:
        .:
        f:clusterRef:
        f:type:
        f:verticalScaling:
          .:
          k:{"componentName":"mongodb"}:
            .:
            f:componentName:
            f:limits:
              .:
              f:cpu:
            f:requests:
              .:
              f:cpu:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-06-26T05:47:45Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:app.kubernetes.io/instance:
        f:ownerReferences:
          .:
          k:{"uid":"24540f4e-9d39-4d3d-b26a-1b2a4328d133"}:
      f:spec:
        f:verticalScaling:
          k:{"componentName":"mongodb"}:
            f:limits:
              f:memory:
            f:requests:
              f:memory:
    Manager:      manager
    Operation:    Update
    Time:         2023-06-26T05:52:19Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:completionTimestamp:
        f:components:
          .:
          f:mongodb:
            .:
            f:phase:
            f:progressDetails:
        f:conditions:
          .:
          k:{"type":"Failed"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Progressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"VerticalScaling"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:lastConfiguration:
          .:
          f:components:
            .:
            f:mongodb:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
        f:phase:
        f:progress:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-06-26T05:52:19Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            cluster-icegqr
    UID:             24540f4e-9d39-4d3d-b26a-1b2a4328d133
  Resource Version:  1155314
  UID:               2f6f44b9-8c71-41aa-94a2-bcfd537ef464
Spec:
  Cluster Ref:  cluster-icegqr
  Type:         VerticalScaling
  Vertical Scaling:
    Component Name:  mongodb
    Limits:
      Cpu:     200m
      Memory:  644245094400m
    Requests:
      Cpu:     200m
      Memory:  644245094400m
Status:
  Cluster Generation:    7
  Completion Timestamp:  2023-06-26T05:52:19Z
  Components:
    Mongodb:
      Phase:  Failed
      Progress Details:
        End Time:    2023-06-26T05:48:35Z
        Message:     Successfully vertical scale: Pod/cluster-icegqr-mongodb-1 in Component: mongodb
        Object Key:  Pod/cluster-icegqr-mongodb-1
        Start Time:  2023-06-26T05:47:46Z
        Status:      Succeed
        End Time:    2023-06-26T05:49:31Z
        Message:     Successfully vertical scale: Pod/cluster-icegqr-mongodb-2 in Component: mongodb
        Object Key:  Pod/cluster-icegqr-mongodb-2
        Start Time:  2023-06-26T05:48:35Z
        Status:      Succeed
        End Time:    2023-06-26T05:52:19Z
        Message:     Failed to vertical scale: Pod/cluster-icegqr-mongodb-0 in Component: mongodb, message: Exec lifecycle hook ([/scripts/replicaset-post-start.sh MONGODB]) for Container "mongodb" in Pod "cluster-icegqr-mongodb-0_default(1fd996ef-d591-4e17-a14b-18c6ec9a9655)" failed - error: command '/scripts/replicaset-post-start.sh MONGODB' exited with 1: MongoNetworkError: connection 3 to 127.0.0.1:27017 closed
MongoServerError: command replSetInitiate requires authentication
, message: "ready\npeer is ready\npeer is ready\npeer is ready\nMongoNetworkError: connection 3 to 127.0.0.1:27017 closed\nMongoServerError: command replSetInitiate requires authentication\n"
        Object Key:  Pod/cluster-icegqr-mongodb-0
        Start Time:  2023-06-26T05:49:31Z
        Status:      Failed
  Conditions:
    Last Transition Time:  2023-06-26T05:47:45Z
    Message:               Start to process the OpsRequest: cluster-icegqr-verticalscaling-q7xxp in Cluster: cluster-icegqr
    Reason:                OpsRequestProgressingStarted
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-06-26T05:47:45Z
    Message:               OpsRequest: cluster-icegqr-verticalscaling-q7xxp is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2023-06-26T05:47:45Z
    Message:               Start to vertical scale resources in Cluster: cluster-icegqr
    Reason:                VerticalScalingStarted
    Status:                True
    Type:                  VerticalScaling
    Last Transition Time:  2023-06-26T05:52:19Z
    Message:               Failed to process OpsRequest: cluster-icegqr-verticalscaling-q7xxp in cluster: cluster-icegqr
    Reason:                OpsRequestFailed
    Status:                False
    Type:                  Failed
  Last Configuration:
    Components:
      Mongodb:
        Limits:
          Cpu:     100m
          Memory:  512Mi
        Requests:
          Cpu:      100m
          Memory:   512Mi
  Phase:            Failed
  Progress:         3/3
  Start Timestamp:  2023-06-26T05:47:45Z
Events:
  Type     Reason                        Age    From                    Message
  ----     ------                        ----   ----                    -------
  Normal   OpsRequestProgressingStarted  9m59s  ops-request-controller  Start to process the OpsRequest: cluster-icegqr-verticalscaling-q7xxp in Cluster: cluster-icegqr
  Normal   ValidateOpsRequestPassed      9m59s  ops-request-controller  OpsRequest: cluster-icegqr-verticalscaling-q7xxp is validated
  Normal   VerticalScalingStarted        9m59s  ops-request-controller  Start to vertical scale resources in Cluster: cluster-icegqr
  Normal   Processing                    9m58s  ops-request-controller  Start to vertical scale: Pod/cluster-icegqr-mongodb-1 in Component: mongodb
  Normal   Succeed                       9m9s   ops-request-controller  Successfully vertical scale: Pod/cluster-icegqr-mongodb-1 in Component: mongodb
  Normal   Processing                    9m9s   ops-request-controller  Start to vertical scale: Pod/cluster-icegqr-mongodb-2 in Component: mongodb
  Normal   Processing                    8m13s  ops-request-controller  Start to vertical scale: Pod/cluster-icegqr-mongodb-0 in Component: mongodb
  Normal   Succeed                       8m13s  ops-request-controller  Successfully vertical scale: Pod/cluster-icegqr-mongodb-2 in Component: mongodb
  Warning  Failed                        5m25s  ops-request-controller  Failed to vertical scale: Pod/cluster-icegqr-mongodb-0 in Component: mongodb, message: Exec lifecycle hook ([/scripts/replicaset-post-start.sh MONGODB]) for Container "mongodb" in Pod "cluster-icegqr-mongodb-0_default(1fd996ef-d591-4e17-a14b-18c6ec9a9655)" failed - error: command '/scripts/replicaset-post-start.sh MONGODB' exited with 1: MongoNetworkError: connection 3 to 127.0.0.1:27017 closed
MongoServerError: command replSetInitiate requires authentication
, message: "ready\npeer is ready\npeer is ready\npeer is ready\nMongoNetworkError: connection 3 to 127.0.0.1:27017 closed\nMongoServerError: command replSetInitiate requires authentication\n"
  Warning  OpsRequestFailed  5m25s  ops-request-controller  Failed to process OpsRequest: cluster-icegqr-verticalscaling-q7xxp in cluster: cluster-icegqr


k get pod
NAME                       READY   STATUS             RESTARTS      AGE
cluster-icegqr-mongodb-0   2/3     CrashLoopBackOff   2 (65s ago)   7m59s
cluster-icegqr-mongodb-1   3/3     Running            0             9m44s
cluster-icegqr-mongodb-2   3/3     Running            0             8m55s

k describe pod cluster-icegqr-mongodb-0
Name:         cluster-icegqr-mongodb-0
Namespace:    default
Priority:     0
Node:         gke-yjtest-default-pool-ce006ea7-7n4n/10.128.15.201
Start Time:   Mon, 26 Jun 2023 13:50:02 +0800
Labels:       app.kubernetes.io/component=mongodb
              app.kubernetes.io/instance=cluster-icegqr
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=mongodb
              app.kubernetes.io/version=mongodb-5.0.14
              apps.kubeblocks.io/component-name=mongodb
              apps.kubeblocks.io/workload-type=Consensus
              controller-revision-hash=cluster-icegqr-mongodb-646d66f5d9
              cs.apps.kubeblocks.io/access-mode=ReadWrite
              kubeblocks.io/role=primary
              statefulset.kubernetes.io/pod-name=cluster-icegqr-mongodb-0
Annotations:  apps.kubeblocks.io/component-replicas: 3
              config.kubeblocks.io/restart-mongodb-config: 549b6f5d68
              cs.apps.kubeblocks.io/leader:
              kubeblocks.io/restart: 2023-06-26T05:39:13Z
Status:       Running
IP:           10.104.0.19
IPs:
  IP:           10.104.0.19
Controlled By:  StatefulSet/cluster-icegqr-mongodb
Containers:
  mongodb:
    Container ID:  containerd://2c96e54c1e3ce7df4fbd6d40d32528b4c7b4d61d75de31eece76237583a74312
    Image:         mongo:5.0.14
    Image ID:      docker.io/library/mongo@sha256:6cf671de7c85493d6dde9aafef383570161809336d6c41f431ac3a692155ece9
    Port:          27017/TCP
    Host Port:     0/TCP
    Command:
      /scripts/replicaset-setup.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 26 Jun 2023 13:54:46 +0800
      Finished:     Mon, 26 Jun 2023 13:56:56 +0800
    Ready:          False
    Restart Count:  2
    Limits:
      cpu:     200m
      memory:  644245094400m
    Requests:
      cpu:     200m
      memory:  644245094400m
    Environment Variables from:
      cluster-icegqr-mongodb-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               cluster-icegqr-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           cluster-icegqr
      KB_COMP_NAME:              mongodb
      KB_CLUSTER_COMP_NAME:      cluster-icegqr-mongodb
      KB_CLUSTER_UID_POSTFIX_8:  4328d133
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MONGODB_ROOT_USER:         <set to the key 'username' in secret 'cluster-icegqr-conn-credential'>  Optional: false
      MONGODB_ROOT_PASSWORD:     <set to the key 'password' in secret 'cluster-icegqr-conn-credential'>  Optional: false
    Mounts:
      /data/mongodb from data (rw)
      /etc/mongodb/keyfile from mongodb-config (rw,path="keyfile")
      /etc/mongodb/mongodb.conf from mongodb-config (rw,path="mongodb.conf")
      /scripts/replicaset-post-start.sh from scripts (rw,path="replicaset-post-start.sh")
      /scripts/replicaset-setup.sh from scripts (rw,path="replicaset-setup.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sl2rm (ro)
  metrics:
    Container ID:  containerd://df67406398937f7b19e888d234e2ec6ad890f2846d17ab2cf682d5075f084f7c
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.0.4
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:57dd82329ef2cd3a93472d184a385a40a39cfb82b9416d292f4bcd409b601774
    Port:          9216/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/conf/metrics-config.yaml
    State:          Running
      Started:      Mon, 26 Jun 2023 13:52:19 +0800
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      cluster-icegqr-mongodb-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               cluster-icegqr-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           cluster-icegqr
      KB_COMP_NAME:              mongodb
      KB_CLUSTER_COMP_NAME:      cluster-icegqr-mongodb
      KB_CLUSTER_UID_POSTFIX_8:  4328d133
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MONGODB_ROOT_USER:         <set to the key 'username' in secret 'cluster-icegqr-conn-credential'>  Optional: false
      MONGODB_ROOT_PASSWORD:     <set to the key 'password' in secret 'cluster-icegqr-conn-credential'>  Optional: false
    Mounts:
      /opt/conf from mongodb-metrics-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sl2rm (ro)
  kb-checkrole:
    Container ID:  containerd://b0d54525a7251655bb2439429002179a49a404324bba21c24cfa86c92425ea7c
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.3-beta.7
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:c2406b6351b9d4e37f4657bcf25d84e38f4b1b5710a93157577811bd7ef67e6b
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      probe
      --app-id
      batch-sdk
      --dapr-http-port
      3501
      --dapr-grpc-port
      50001
      --app-protocol
      http
      --log-level
      info
      --config
      /config/probe/config.yaml
      --components-path
      /config/probe/components
    State:          Running
      Started:      Mon, 26 Jun 2023 13:52:19 +0800
    Ready:          True
    Restart Count:  0
    Readiness:      exec [curl -X POST --max-time 1 --fail-with-body --silent -H Content-ComponentDefRef: application/json http://localhost:3501/v1.0/bindings/mongodb -d {"operation": "checkRole", "metadata":{"sql":""}}] delay=0s timeout=1s period=2s #success=1 #failure=3
    Startup:        tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      cluster-icegqr-mongodb-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                cluster-icegqr-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            cluster-icegqr
      KB_COMP_NAME:               mongodb
      KB_CLUSTER_COMP_NAME:       cluster-icegqr-mongodb
      KB_CLUSTER_UID_POSTFIX_8:   4328d133
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'cluster-icegqr-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'cluster-icegqr-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            27017
      KB_SERVICE_ROLES:           {"primary":"ReadWrite","secondary":"Readonly"}
      KB_SERVICE_CHARACTER_TYPE:  mongodb
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sl2rm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-cluster-icegqr-mongodb-0
    ReadOnly:   false
  mongodb-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      cluster-icegqr-mongodb-mongodb-config
    Optional:  false
  mongodb-metrics-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      cluster-icegqr-mongodb-mongodb-metrics-config
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      cluster-icegqr-mongodb-mongodb-scripts
    Optional:  false
  kube-api-access-sl2rm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason               Age                   From               Message
  ----     ------               ----                  ----               -------
  Normal   Scheduled            8m12s                 default-scheduler  Successfully assigned default/cluster-icegqr-mongodb-0 to gke-yjtest-default-pool-ce006ea7-7n4n
  Normal   Pulled               5m55s                 kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.0.4" already present on machine
  Normal   Created              5m55s                 kubelet            Created container metrics
  Normal   Pulled               5m55s                 kubelet            Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.3-beta.7" already present on machine
  Normal   Created              5m55s                 kubelet            Created container kb-checkrole
  Normal   Started              5m55s                 kubelet            Started container kb-checkrole
  Normal   Started              5m55s                 kubelet            Started container metrics
  Normal   checkRole            5m38s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"","role":"secondary"}
  Normal   checkRole            5m28s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            5m16s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            5m4s                  sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            4m52s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            4m38s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            4m26s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            4m14s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            4m2s                  sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            3m48s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            3m45s                 sqlchannel         {"event":"Failed","message":"server selection error: context deadline exceeded, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: dial tcp 127.0.0.1:27017: connect: connection refused }, ] }","operation":"checkRole","originalRole":"primary"}
  Normal   Pulled               3m28s (x3 over 8m3s)  kubelet            Container image "mongo:5.0.14" already present on machine
  Normal   Created              3m28s (x3 over 8m3s)  kubelet            Created container mongodb
  Normal   Started              3m28s (x3 over 8m3s)  kubelet            Started container mongodb
  Normal   checkRole            3m10s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            3m2s                  sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            2m48s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            2m36s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            2m24s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole            2m12s                 sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole            2m                    sqlchannel         {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   Killing              108s (x3 over 6m25s)  kubelet            FailedPostStartHook
  Warning  FailedPostStartHook  108s (x3 over 6m25s)  kubelet            Exec lifecycle hook ([/scripts/replicaset-post-start.sh MONGODB]) for Container "mongodb" in Pod "cluster-icegqr-mongodb-0_default(1fd996ef-d591-4e17-a14b-18c6ec9a9655)" failed - error: command '/scripts/replicaset-post-start.sh MONGODB' exited with 1: MongoNetworkError: connection 3 to 127.0.0.1:27017 closed
MongoServerError: command replSetInitiate requires authentication
, message: "ready\npeer is ready\npeer is ready\npeer is ready\nMongoNetworkError: connection 3 to 127.0.0.1:27017 closed\nMongoServerError: command replSetInitiate requires authentication\n"
  Normal   checkRole  108s                 sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole  98s                  sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole  82s                  sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole  77s                  sqlchannel  {"event":"Failed","message":"server selection error: context deadline exceeded, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: dial tcp 127.0.0.1:27017: connect: connection refused }, ] }","operation":"checkRole","originalRole":"primary"}
  Warning  BackOff    76s (x5 over 3m45s)  kubelet     Back-off restarting failed container
  Normal   checkRole  40s                  sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole  30s                  sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
  Normal   checkRole  18s                  sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"primary","role":"secondary"}
  Normal   checkRole  6s                   sqlchannel  {"event":"Success","operation":"checkRole","originalRole":"secondary","role":"primary"}
@ahjing99 ahjing99 added the kind/bug Something isn't working label Jun 26, 2023
@ahjing99 ahjing99 added this to the Release 0.6.0 milestone Jun 26, 2023
@ahjing99 ahjing99 changed the title [BUG]vscale ops failed for pod crash [BUG]MongoDB vscale ops failed for pod crash Jun 26, 2023
This was linked to pull requests Jun 26, 2023
fix: mongodb post start script #3954
Merged
fix: mongodb post start script #3956
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xuriwuyun xuriwuyun

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.6.0
3 participants
@xuriwuyun @ahjing99 @apecloud-bot