Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]MongoDB pod is always Terminating when stop cluster #3950

Closed
ahjing99 opened this issue Jun 26, 2023 · 1 comment · Fixed by #3971, #3972 or #3990
Closed

[BUG]MongoDB pod is always Terminating when stop cluster #3950

ahjing99 opened this issue Jun 26, 2023 · 1 comment · Fixed by #3971, #3972 or #3990
Assignees
@xuriwuyun
Labels
bug kind/bug Something isn't working
Milestone
Release 0.6.0

Comments

@ahjing99
Copy link
Collaborator

ahjing99 commented Jun 26, 2023 •
edited
Loading

➜ ~ kbcli version
Kubernetes: v1.25.10-eks-c12679a
KubeBlocks: 0.5.3-beta.7
kbcli: 0.5.3-beta.7

  1. Create and then stop the cluster
 `kbcli cluster create mongocluster              --termination-policy=Halt              --monitor=false --enable-all-logs=false --cluster-definition=mongodb --set type=mongodb,cpu=100m,memory=0.5Gi,replicas=3,storage=1Gi  --namespace default `

Info: --cluster-version is not specified, ClusterVersion mongodb-5.0.14 is applied by default
Cluster mongocluster created

 `echo mongocluster | kbcli cluster stop mongocluster --namespace default `

Please type the name again(separate with white space when more than one): mongocluster
OpsRequest mongocluster-stop-hwqf6 created successfully, you can view the progress:
	kbcli cluster describe-ops mongocluster-stop-hwqf6 -n default
  1. Pod is always Terminating
➜  ~ kbcli cluster describe mongocluster
Name: mongocluster	 Created Time: Jun 26,2023 14:01 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION          STATUS     TERMINATION-POLICY
default     mongodb              mongodb-5.0.14   Stopping   Halt

Endpoints:
COMPONENT   MODE        INTERNAL                                               EXTERNAL
mongodb     ReadWrite   mongocluster-mongodb.default.svc.cluster.local:27017   <none>

Topology:
COMPONENT   INSTANCE                 ROLE     STATUS    AZ                NODE                                                       CREATED-TIME
mongodb     mongocluster-mongodb-0   <none>   Pending   cn-northwest-1a   ip-10-0-1-229.cn-northwest-1.compute.internal/10.0.1.229   Jun 26,2023 14:08 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mongodb     false       100m / 100m          512Mi / 512Mi           data:1Gi       gp3

Images:
COMPONENT   TYPE      IMAGE
mongodb     mongodb   mongo:5.0.14

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default mongocluster

➜  ~ k get ops
NAME                      TYPE   CLUSTER        STATUS    PROGRESS   AGE
mongocluster-stop-hwqf6   Stop   mongocluster   Running   2/3        17m
➜  ~ k get pod
NAME                     READY   STATUS        RESTARTS   AGE
mongocluster-mongodb-0   0/3     Terminating   0          18m

➜  ~ k describe ops mongocluster-stop-hwqf6
Name:         mongocluster-stop-hwqf6
Namespace:    default
Labels:       app.kubernetes.io/instance=mongocluster
Annotations:  kubeblocks.io/reconcile: 2023-06-26T06:09:33.243888206Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2023-06-26T06:09:00Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  mongocluster-stop-
  Generation:     2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
      f:spec:
        .:
        f:clusterRef:
        f:type:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-06-26T06:09:00Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:app.kubernetes.io/instance:
        f:ownerReferences:
          .:
          k:{"uid":"105f28c7-fe9d-4d68-a120-e31ae366ec67"}:
    Manager:      manager
    Operation:    Update
    Time:         2023-06-26T06:09:33Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:components:
          .:
          f:mongodb:
            .:
            f:phase:
            f:progressDetails:
        f:conditions:
          .:
          k:{"type":"Progressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Stopping"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:lastConfiguration:
          .:
          f:components:
            .:
            f:mongodb:
              .:
              f:replicas:
              f:targetResources:
                .:
                f:pods:
        f:phase:
        f:progress:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-06-26T06:09:33Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            mongocluster
    UID:             105f28c7-fe9d-4d68-a120-e31ae366ec67
  Resource Version:  7666
  UID:               8556cf44-0ccc-490c-9814-5a70c7ad198d
Spec:
  Cluster Ref:  mongocluster
  Type:         Stop
Status:
  Cluster Generation:  3
  Components:
    Mongodb:
      Phase:  Updating
      Progress Details:
        Message:     Start to delete pod: Pod/mongocluster-mongodb-0 in Component: mongodb
        Object Key:  Pod/mongocluster-mongodb-0
        Start Time:  2023-06-26T06:09:01Z
        Status:      Processing
        End Time:    2023-06-26T06:09:33Z
        Message:     Successfully deleted pod: Pod/mongocluster-mongodb-1 in Component: mongodb
        Object Key:  Pod/mongocluster-mongodb-1
        Start Time:  2023-06-26T06:09:01Z
        Status:      Succeed
        End Time:    2023-06-26T06:09:33Z
        Message:     Successfully deleted pod: Pod/mongocluster-mongodb-2 in Component: mongodb
        Object Key:  Pod/mongocluster-mongodb-2
        Start Time:  2023-06-26T06:09:01Z
        Status:      Succeed
  Conditions:
    Last Transition Time:  2023-06-26T06:09:00Z
    Message:               Start to process the OpsRequest: mongocluster-stop-hwqf6 in Cluster: mongocluster
    Reason:                OpsRequestProgressingStarted
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-06-26T06:09:01Z
    Message:               OpsRequest: mongocluster-stop-hwqf6 is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2023-06-26T06:09:01Z
    Message:               Start to stop the Cluster: mongocluster
    Reason:                StopStarted
    Status:                True
    Type:                  Stopping
  Last Configuration:
    Components:
      Mongodb:
        Replicas:  3
        Target Resources:
          Pods:
            mongocluster-mongodb-2
            mongocluster-mongodb-0
            mongocluster-mongodb-1
  Phase:            Running
  Progress:         2/3
  Start Timestamp:  2023-06-26T06:09:01Z
Events:
  Type    Reason                        Age                From                    Message
  ----    ------                        ----               ----                    -------
  Normal  OpsRequestProgressingStarted  18m                ops-request-controller  Start to process the OpsRequest: mongocluster-stop-hwqf6 in Cluster: mongocluster
  Normal  ValidateOpsRequestPassed      18m                ops-request-controller  OpsRequest: mongocluster-stop-hwqf6 is validated
  Normal  StopStarted                   18m                ops-request-controller  Start to stop the Cluster: mongocluster
  Normal  Processing                    18m                ops-request-controller  Start to delete pod: Pod/mongocluster-mongodb-0 in Component: mongodb
  Normal  Processing                    18m                ops-request-controller  Start to delete pod: Pod/mongocluster-mongodb-1 in Component: mongodb
  Normal  Processing                    18m                ops-request-controller  Start to delete pod: Pod/mongocluster-mongodb-2 in Component: mongodb
  Normal  Succeed                       17m (x2 over 17m)  ops-request-controller  Successfully deleted pod: Pod/mongocluster-mongodb-2 in Component: mongodb
  Normal  Succeed                       17m (x2 over 17m)  ops-request-controller  Successfully deleted pod: Pod/mongocluster-mongodb-1 in Component: mongodb

➜  ~ k describe pod mongocluster-mongodb-0
Name:                      mongocluster-mongodb-0
Namespace:                 default
Priority:                  0
Node:                      ip-10-0-1-229.cn-northwest-1.compute.internal/10.0.1.229
Start Time:                Mon, 26 Jun 2023 14:08:21 +0800
Labels:                    app.kubernetes.io/component=mongodb
                           app.kubernetes.io/instance=mongocluster
                           app.kubernetes.io/managed-by=kubeblocks
                           app.kubernetes.io/name=mongodb
                           app.kubernetes.io/version=mongodb-5.0.14
                           apps.kubeblocks.io/component-name=mongodb
                           apps.kubeblocks.io/workload-type=Consensus
                           controller-revision-hash=mongocluster-mongodb-76fbdf56b5
                           statefulset.kubernetes.io/pod-name=mongocluster-mongodb-0
Annotations:               apps.kubeblocks.io/component-replicas: 0
                           cs.apps.kubeblocks.io/leader:
Status:                    Terminating (lasts 18m)
Termination Grace Period:  30s
IP:
IPs:                       <none>
Controlled By:             StatefulSet/mongocluster-mongodb
Containers:
  mongodb:
    Container ID:
    Image:         mongo:5.0.14
    Image ID:
    Port:          27017/TCP
    Host Port:     0/TCP
    Command:
      /scripts/replicaset-setup.sh
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  512Mi
    Environment Variables from:
      mongocluster-mongodb-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mongocluster-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mongocluster
      KB_COMP_NAME:              mongodb
      KB_CLUSTER_COMP_NAME:      mongocluster-mongodb
      KB_CLUSTER_UID_POSTFIX_8:  e366ec67
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MONGODB_ROOT_USER:         <set to the key 'username' in secret 'mongocluster-conn-credential'>  Optional: false
      MONGODB_ROOT_PASSWORD:     <set to the key 'password' in secret 'mongocluster-conn-credential'>  Optional: false
    Mounts:
      /data/mongodb from data (rw)
      /etc/mongodb/keyfile from mongodb-config (rw,path="keyfile")
      /etc/mongodb/mongodb.conf from mongodb-config (rw,path="mongodb.conf")
      /scripts/replicaset-post-start.sh from scripts (rw,path="replicaset-post-start.sh")
      /scripts/replicaset-setup.sh from scripts (rw,path="replicaset-setup.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4prwn (ro)
  metrics:
    Container ID:
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.0.4
    Image ID:
    Port:          9216/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/conf/metrics-config.yaml
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      mongocluster-mongodb-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mongocluster-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mongocluster
      KB_COMP_NAME:              mongodb
      KB_CLUSTER_COMP_NAME:      mongocluster-mongodb
      KB_CLUSTER_UID_POSTFIX_8:  e366ec67
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MONGODB_ROOT_USER:         <set to the key 'username' in secret 'mongocluster-conn-credential'>  Optional: false
      MONGODB_ROOT_PASSWORD:     <set to the key 'password' in secret 'mongocluster-conn-credential'>  Optional: false
    Mounts:
      /opt/conf from mongodb-metrics-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4prwn (ro)
  kb-checkrole:
    Container ID:
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.5.3-beta.7
    Image ID:
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      probe
      --app-id
      batch-sdk
      --dapr-http-port
      3501
      --dapr-grpc-port
      50001
      --app-protocol
      http
      --log-level
      info
      --config
      /config/probe/config.yaml
      --components-path
      /config/probe/components
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Readiness:      exec [curl -X POST --max-time 1 --fail-with-body --silent -H Content-ComponentDefRef: application/json http://localhost:3501/v1.0/bindings/mongodb -d {"operation": "checkRole", "metadata":{"sql":""}}] delay=0s timeout=1s period=2s #success=1 #failure=3
    Startup:        tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      mongocluster-mongodb-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                mongocluster-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            mongocluster
      KB_COMP_NAME:               mongodb
      KB_CLUSTER_COMP_NAME:       mongocluster-mongodb
      KB_CLUSTER_UID_POSTFIX_8:   e366ec67
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'mongocluster-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'mongocluster-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            27017
      KB_SERVICE_ROLES:           {"primary":"ReadWrite","secondary":"Readonly"}
      KB_SERVICE_CHARACTER_TYPE:  mongodb
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4prwn (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-mongocluster-mongodb-0
    ReadOnly:   false
  mongodb-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongocluster-mongodb-mongodb-config
    Optional:  false
  mongodb-metrics-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongocluster-mongodb-mongodb-metrics-config
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongocluster-mongodb-mongodb-scripts
    Optional:  false
  kube-api-access-4prwn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  19m   default-scheduler  Successfully assigned default/mongocluster-mongodb-0 to ip-10-0-1-229.cn-northwest-1.compute.internal
  Normal  Pulled     19m   kubelet            Container image "mongo:5.0.14" already present on machine
  Normal  Created    19m   kubelet            Created container mongodb
  Normal  Started    19m   kubelet            Started container mongodb


➜  ~ k logs mongocluster-mongodb-0
Defaulted container "mongodb" out of: mongodb, metrics, kb-checkrole
Error from server (BadRequest): container "mongodb" in pod "mongocluster-mongodb-0" is waiting to start: ContainerCreating

➜  ~ k describe cluster mongocluster
Name:         mongocluster
Namespace:    default
Labels:       app.kubernetes.io/instance=mongocluster
              clusterdefinition.kubeblocks.io/name=mongodb
              clusterversion.kubeblocks.io/name=mongodb-5.0.14
Annotations:  kubeblocks.io/ops-request: [{"name":"mongocluster-stop-hwqf6","type":"Stop"}]
              kubeblocks.io/snapshot-for-start: {"mongodb":3}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-06-26T06:01:27Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  3
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:app.kubernetes.io/instance:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mongodb"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-06-26T06:07:17Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/ops-request:
          f:kubeblocks.io/snapshot-for-start:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"mongodb"}:
            f:replicas:
    Manager:      manager
    Operation:    Update
    Time:         2023-06-26T06:09:01Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mongodb:
            .:
            f:consensusSetStatus:
              .:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:phase:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-06-26T06:09:02Z
  Resource Version:  7473
  UID:               105f28c7-fe9d-4d68-a120-e31ae366ec67
Spec:
  Affinity:
    Pod Anti Affinity:     Preferred
    Tenancy:               SharedNode
  Cluster Definition Ref:  mongodb
  Cluster Version Ref:     mongodb-5.0.14
  Component Specs:
    Component Def Ref:  mongodb
    Monitor:            false
    Name:               mongodb
    Replicas:           0
    Resources:
      Limits:
        Cpu:     100m
        Memory:  512Mi
      Requests:
        Cpu:               100m
        Memory:            512Mi
    Service Account Name:  kb-sa-mongocluster
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:   1Gi
  Termination Policy:  Halt
Status:
  Cluster Def Generation:  2
  Components:
    Mongodb:
      Consensus Set Status:
        Leader:
          Access Mode:  None
          Name:
          Pod:          Unknown
      Phase:            Updating
  Conditions:
    Last Transition Time:  2023-06-26T06:09:01Z
    Message:               Stop opsRequest: mongocluster-stop-hwqf6 is processing
    Reason:                Stopping
    Status:                False
    Type:                  LatestOpsRequestProcessed
    Last Transition Time:  2023-06-26T06:01:27Z
    Message:               The operator has started the provisioning of Cluster: mongocluster
    Observed Generation:   3
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-06-26T06:09:02Z
    Message:               Successfully applied for resources
    Observed Generation:   3
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-06-26T06:07:55Z
    Message:               pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-06-26T06:07:55Z
    Message:               pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     3
  Phase:                   Updating
Events:
  Type     Reason                 Age                From                    Message
  ----     ------                 ----               ----                    -------
  Normal   AllReplicasReady       23m                cluster-controller      all pods of components are ready, waiting for the probe detection successful
  Normal   Running                23m                cluster-controller      Cluster: mongocluster is ready, current phase is Running
  Normal   ClusterReady           23m                cluster-controller      Cluster: mongocluster is ready, current phase is Running
  Warning  ApplyResourcesFailed   21m (x2 over 23m)  cluster-controller      Operation cannot be fulfilled on pods "mongocluster-mongodb-1": the object has been modified; please apply your changes to the latest version and try again
  Warning  ReplicasNotReady       21m                cluster-controller      pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady     21m                cluster-controller      pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components
  Warning  ApplyResourcesFailed   21m                cluster-controller      Operation cannot be fulfilled on pods "mongocluster-mongodb-0": the object has been modified; please apply your changes to the latest version and try again
  Normal   PreCheckSucceed        20m (x2 over 28m)  cluster-controller      The operator has started the provisioning of Cluster: mongocluster
  Normal   Stop                   20m                ops-request-controller  Start to process the Stop opsRequest "mongocluster-stop-hwqf6" in Cluster: mongocluster
  Normal   HorizontalScale        20m (x2 over 20m)  cluster-controller      Start horizontal scale component mongodb from 3 to 0
  Normal   ApplyResourcesSucceed  20m (x8 over 28m)  cluster-controller      Successfully applied for resources
  Warning  ApplyResourcesFailed   20m (x3 over 28m)  cluster-controller      Operation cannot be fulfilled on pods "mongocluster-mongodb-2": the object has been modified; please apply your changes to the latest version and try again
@ahjing99 ahjing99 added the kind/bug Something isn't working label Jun 26, 2023
@ahjing99 ahjing99 added this to the Release 0.6.0 milestone Jun 26, 2023
@ahjing99
Copy link
Collaborator Author

k logs kubeblocks-6cc677997b-4rf5v -n kb-system > kb.txt
kb.txt

This was linked to pull requests Jun 27, 2023
fix: mongodb pod terminating takes too long time #3971
Merged
fix: mongodb pod terminating take too long time #3972
Merged
@xuriwuyun xuriwuyun linked a pull request Jun 27, 2023 that will close this issue
fix: MongoDB does not receive signals when terminating pods. #3990
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xuriwuyun xuriwuyun

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.6.0
3 participants
@xuriwuyun @ahjing99 @apecloud-bot