Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] delete cluster pod always Terminating with csi-hostpath-driver on k3s/kind/minikube #8233

Open
JashBook opened this issue Sep 30, 2024 · 1 comment
Open

[BUG] delete cluster pod always Terminating with csi-hostpath-driver on k3s/kind/minikube #8233

JashBook opened this issue Sep 30, 2024 · 1 comment
Assignees
@free6om
Labels
kind/bug Something isn't working
Milestone
Release 0.9.2

Comments

@JashBook
Copy link
Collaborator

JashBook commented Sep 30, 2024 •
edited
Loading

Describe the bug

Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:04:48.280621419 +0000 UTC m=+4797.843662028 (durationBeforeRetry 1m4s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list

kbcli version                                                                                
Kubernetes: v1.30.4+k3s1
KubeBlocks: 0.9.1-beta.31
kbcli: 0.9.1-beta.14

To Reproduce
Steps to reproduce the behavior:

  1. install kubeblocks
  2. enable csi-hostpath-driver
kbcli addon enable csi-hostpath-driver
  1. create cluster
kbcli cluster create  kafka kafka-dukzlp                 --mode='combined'                 --cpu=0.5                 --memory=0.5                 --storage=1                 --availability-policy=none --termination-policy=WipeOut --version=kafka-3.3.2  --storage-enable=true                 --meta-storage=1 --replicas=1

kubectl get pod 
NAME                            READY   STATUS    RESTARTS   AGE
kafka-dukzlp-kafka-combine-0    2/2     Running   0          46s
kafka-dukzlp-kafka-exporter-0   1/1     Running   0          12s
➜  ~ kubectl get cluster 
NAME           CLUSTER-DEFINITION   VERSION       TERMINATION-POLICY   STATUS    AGE
kafka-dukzlp   kafka                kafka-3.3.2   WipeOut       Running   49s
  1. delete cluster
kbcli cluster delete kafka-dukzlp --auto-approve
  1. See error
kubectl get pod 
NAME                           READY   STATUS        RESTARTS   AGE
kafka-dukzlp-kafka-combine-0   0/2     Terminating   0          3m42s

describe its

describe its kafka-dukzlp-kafka-combine 
Name:         kafka-dukzlp-kafka-combine
Namespace:    default
Labels:       app.kubernetes.io/component=kafka-combine
              app.kubernetes.io/instance=kafka-dukzlp
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=kafka-combine
              apps.kubeblocks.io/cluster-uid=411c0cc0-e2f0-4735-92e5-63ecfecc4234
              apps.kubeblocks.io/component-name=kafka-combine
              clusterdefinition.kubeblocks.io/name=kafka
              clusterversion.kubeblocks.io/name=kafka-3.3.2
              componentdefinition.kubeblocks.io/name=kafka-combine
              helm.sh/chart=kafka-cluster-0.9.0
Annotations:  app.kubernetes.io/component: kafka-combine
              apps.kubeblocks.io/service-version: 3.3.2
              config.kubeblocks.io/tpl-kafka-configuration-tpl: kafka-dukzlp-kafka-combine-kafka-configuration-tpl
              config.kubeblocks.io/tpl-kafka-jmx-configuration-tpl: kafka-dukzlp-kafka-combine-kafka-jmx-configuration-tpl
              config.kubeblocks.io/tpl-kafka-scripts-tools-tpl: kafka-dukzlp-kafka-combine-kafka-scripts-tools-tpl
              config.kubeblocks.io/tpl-kafka-scripts-tpl: kafka-dukzlp-kafka-combine-kafka-scripts-tpl
              kubeblocks.io/extra-env:
                {"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":...
              kubeblocks.io/generation: 3
              monitor.kubeblocks.io/path.headless.its: /metrics
              monitor.kubeblocks.io/port.headless.its: 5556
              monitor.kubeblocks.io/scheme.headless.its: http
              monitor.kubeblocks.io/scrape.headless.its: true
API Version:  workloads.kubeblocks.io/v1alpha1
Kind:         InstanceSet
Metadata:
  Creation Timestamp:             2024-09-30T03:00:37Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2024-09-30T03:02:38Z
  Finalizers:
    instanceset.workloads.kubeblocks.io/finalizer
  Generation:  6
  Owner References:
    API Version:           apps.kubeblocks.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Component
    Name:                  kafka-dukzlp-kafka-combine
    UID:                   1a9d3526-66f9-4f26-b773-8f63e67758c4
  Resource Version:        14385
  UID:                     5c6a4d9a-0a2e-4a20-b88c-6f606b3c6517
Spec:
  Default Template Ordinals:
  Member Update Strategy:               BestEffortParallel
  Min Ready Seconds:                    0
  Parallel Pod Management Concurrency:  100%
  Pod Management Policy:                Parallel
  Pod Update Policy:                    PreferInPlace
  Replicas:                             1
  Selector:
    Match Labels:
      app.kubernetes.io/instance:         kafka-dukzlp
      app.kubernetes.io/managed-by:       kubeblocks
      app.kubernetes.io/name:             kafka-combine
      apps.kubeblocks.io/component-name:  kafka-combine
  Template:
    Metadata:
      Annotations:
        apps.kubeblocks.io/component-replicas:  1
      Labels:
        app.kubernetes.io/component:             kafka-combine
        app.kubernetes.io/instance:              kafka-dukzlp
        app.kubernetes.io/managed-by:            kubeblocks
        app.kubernetes.io/name:                  kafka-combine
        app.kubernetes.io/version:               kafka-combine
        apps.kubeblocks.io/cluster-uid:          411c0cc0-e2f0-4735-92e5-63ecfecc4234
        apps.kubeblocks.io/component-name:       kafka-combine
        clusterdefinition.kubeblocks.io/name:    kafka
        clusterversion.kubeblocks.io/name:       kafka-3.3.2
        componentdefinition.kubeblocks.io/name:  kafka-combine
        helm.sh/chart:                           kafka-cluster-0.9.0
    Spec:
      Affinity:
        Node Affinity:
          Preferred During Scheduling Ignored During Execution:
            Preference:
              Match Expressions:
                Key:       kb-data
                Operator:  In
                Values:
                  true
            Weight:  100
        Pod Anti Affinity:
      Containers:
        Command:
          /scripts/kafka-server-setup.sh
        Env:
          Name:  SUPER_USER
          Value From:
            Secret Key Ref:
              Key:   username
              Name:  kafka-dukzlp-kafka-combine-account-admin
          Name:      KB_POD_NAME
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   metadata.name
          Name:             KB_POD_UID
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   metadata.uid
          Name:             KB_NAMESPACE
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   metadata.namespace
          Name:             KB_SA_NAME
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   spec.serviceAccountName
          Name:             KB_NODENAME
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   spec.nodeName
          Name:             KB_HOST_IP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.hostIP
          Name:             KB_POD_IP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIP
          Name:             KB_POD_IPS
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIPs
          Name:             KB_HOSTIP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.hostIP
          Name:             KB_PODIP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIP
          Name:             KB_PODIPS
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIPs
          Name:             KB_POD_FQDN
          Value:            $(KB_POD_NAME).kafka-dukzlp-kafka-combine-headless.$(KB_NAMESPACE).svc
          Name:             BITNAMI_DEBUG
          Value:            true
          Name:             MY_POD_IP
          Value:            $(KB_PODIP)
          Name:             MY_POD_NAME
          Value:            $(KB_POD_NAME)
          Name:             KAFKA_ENABLE_KRAFT
          Value:            yes
          Name:             KAFKA_CFG_PROCESS_ROLES
          Value:            broker,controller
          Name:             KAFKA_CFG_CONTROLLER_LISTENER_NAMES
          Value:            CONTROLLER
          Name:             KAFKA_CFG_INTER_BROKER_LISTENER_NAME
          Value:            INTERNAL
          Name:             KAFKA_CFG_LISTENERS
          Value:            CONTROLLER://:9093,INTERNAL://:9094,CLIENT://:9092
          Name:             KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP
          Value:            CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT
          Name:             KAFKA_CFG_ADVERTISED_LISTENERS
          Value:            INTERNAL://$(KB_POD_IP):9094,CLIENT://$(KB_POD_IP):9092
          Name:             KAFKA_CFG_INITIAL_BROKER_REGISTRATION_TIMEOUT_MS
          Value:            240000
          Name:             ALLOW_PLAINTEXT_LISTENER
          Value:            yes
          Name:             JMX_PORT
          Value:            5555
          Name:             KAFKA_VOLUME_DIR
          Value:            /bitnami/kafka
          Name:             KAFKA_CFG_METADATA_LOG_DIR
          Value:            /bitnami/kafka/metadata
          Name:             KAFKA_LOG_DIR
          Value:            /bitnami/kafka/data
          Name:             KAFKA_HEAP_OPTS
          Value:            -XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64
          Name:             SERVER_PROP_FILE
          Value:            /scripts/server.properties
          Name:             KAFKA_KRAFT_CLUSTER_ID
          Value:            $(KB_CLUSTER_UID)
          Name:             KAFKA_CFG_SUPER_USERS
          Value:            User:$(SUPER_USER)
          Name:             KB_KAFKA_SASL_CONFIG_PATH
          Value:            /tools/server-jaas.properties
        Env From:
          Config Map Ref:
            Name:           kafka-dukzlp-kafka-combine-env
            Optional:       false
        Image:              docker.io/apecloud/kafka:3.3.2-debian-11-r54
        Image Pull Policy:  IfNotPresent
        Liveness Probe:
          Failure Threshold:      3
          Initial Delay Seconds:  10
          Period Seconds:         10
          Success Threshold:      1
          Tcp Socket:
            Port:           kafka-ctrlr
          Timeout Seconds:  5
        Name:               kafka
        Ports:
          Container Port:  9092
          Name:            kafka-client
          Protocol:        TCP
          Container Port:  9093
          Name:            kafka-ctrlr
          Protocol:        TCP
          Container Port:  9094
          Name:            kafka-internal
          Protocol:        TCP
        Resources:
          Limits:
            Cpu:     500m
            Memory:  512Mi
          Requests:
            Cpu:     500m
            Memory:  512Mi
        Security Context:
          Allow Privilege Escalation:  false
          Run As Non Root:             true
          Run As User:                 1001
        Startup Probe:
          Failure Threshold:      30
          Initial Delay Seconds:  5
          Period Seconds:         10
          Success Threshold:      1
          Tcp Socket:
            Port:           kafka-ctrlr
          Timeout Seconds:  5
        Volume Mounts:
          Mount Path:  /bitnami/kafka
          Name:        data
          Mount Path:  /bitnami/kafka/metadata
          Name:        metadata
          Mount Path:  /scripts/kafka-server-setup.sh
          Name:        scripts
          Sub Path:    kafka-server-setup.sh
          Mount Path:  /scripts/server.properties
          Name:        kafka-config
          Sub Path:    server.properties
          Mount Path:  /tools/client-ssl.properties
          Name:        tools
          Sub Path:    client-ssl.properties
          Mount Path:  /tools/server-jaas.properties
          Name:        tools
          Sub Path:    server-jaas.properties
        Args:
          -XX:MaxRAMPercentage=100
          -XshowSettings:vm
          -jar
          jmx_prometheus_httpserver.jar
          5556
          /etc/jmx-kafka/jmx-kafka-prometheus.yml
        Command:
          java
        Env:
          Name:  SUPER_USER
          Value From:
            Secret Key Ref:
              Key:   username
              Name:  kafka-dukzlp-kafka-combine-account-admin
          Name:      KB_POD_NAME
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   metadata.name
          Name:             KB_POD_UID
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   metadata.uid
          Name:             KB_NAMESPACE
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   metadata.namespace
          Name:             KB_SA_NAME
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   spec.serviceAccountName
          Name:             KB_NODENAME
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   spec.nodeName
          Name:             KB_HOST_IP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.hostIP
          Name:             KB_POD_IP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIP
          Name:             KB_POD_IPS
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIPs
          Name:             KB_HOSTIP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.hostIP
          Name:             KB_PODIP
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIP
          Name:             KB_PODIPS
          Value From:
            Field Ref:
              API Version:  v1
              Field Path:   status.podIPs
          Name:             KB_POD_FQDN
          Value:            $(KB_POD_NAME).kafka-dukzlp-kafka-combine-headless.$(KB_NAMESPACE).svc
          Name:             SERVICE_PORT
          Value:            5556
        Env From:
          Config Map Ref:
            Name:           kafka-dukzlp-kafka-combine-env
            Optional:       false
        Image:              docker.io/apecloud/jmx-exporter:0.18.0-debian-11-r20
        Image Pull Policy:  IfNotPresent
        Name:               jmx-exporter
        Ports:
          Container Port:  5556
          Name:            metrics
          Protocol:        TCP
        Resources:
          Limits:
            Cpu:     0
            Memory:  0
        Security Context:
          Run As Non Root:  true
          Run As User:      1001
        Volume Mounts:
          Mount Path:  /etc/jmx-kafka
          Name:        jmx-config
      Security Context:
        Fs Group:  1001
      Tolerations:
        Effect:    NoSchedule
        Key:       kb-data
        Operator:  Equal
        Value:     true
      Volumes:
        Config Map:
          Default Mode:  292
          Name:          kafka-dukzlp-kafka-combine-kafka-configuration-tpl
        Name:            kafka-config
        Config Map:
          Default Mode:  292
          Name:          kafka-dukzlp-kafka-combine-kafka-jmx-configuration-tpl
        Name:            jmx-config
        Config Map:
          Default Mode:  493
          Name:          kafka-dukzlp-kafka-combine-kafka-scripts-tools-tpl
        Name:            tools
        Config Map:
          Default Mode:  493
          Name:          kafka-dukzlp-kafka-combine-kafka-scripts-tpl
        Name:            scripts
        Empty Dir:
        Name:  data
        Empty Dir:
        Name:  metadata
  Update Strategy:
    Member Update Strategy:  BestEffortParallel
  Volume Claim Templates:
    Metadata:
      Labels:
        apps.kubeblocks.io/vct-name:  data
      Name:                           data
    Spec:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:  1Gi
    Status:
    Metadata:
      Labels:
        apps.kubeblocks.io/vct-name:  metadata
      Name:                           metadata
    Spec:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:  1Gi
    Status:
Status:
  Available Replicas:  1
  Conditions:
    Last Transition Time:  2024-09-30T03:01:10Z
    Message:               
    Observed Generation:   4
    Reason:                Ready
    Status:                True
    Type:                  InstanceReady
    Last Transition Time:  2024-09-30T03:01:10Z
    Message:               
    Observed Generation:   4
    Reason:                Available
    Status:                True
    Type:                  InstanceAvailable
  Current Replicas:        1
  Current Revision:        7cc7f549b6
  Current Revisions:
    kafka-dukzlp-kafka-combine-0:  7cc7f549b6
  Init Replicas:                   0
  Observed Generation:             6
  Ready Replicas:                  1
  Replicas:                        1
  Update Revision:                 7cc7f549b6
  Update Revisions:
    kafka-dukzlp-kafka-combine-0:  7cc7f549b6
Events:
  Type    Reason            Age   From                     Message
  ----    ------            ----  ----                     -------
  Normal  SuccessfulUpdate  49m   instance-set-controller  update InstanceSet kafka-dukzlp-kafka-combine in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulCreate  49m   instance-set-controller  create ConfigMap kafka-dukzlp-kafka-combine-rsm-env in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulCreate  49m   instance-set-controller  create Service kafka-dukzlp-kafka-combine-headless in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulCreate  49m   instance-set-controller  create PersistentVolumeClaim metadata-kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulCreate  49m   instance-set-controller  create PersistentVolumeClaim data-kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulCreate  49m   instance-set-controller  create Pod kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulDelete  47m   instance-set-controller  delete ConfigMap kafka-dukzlp-kafka-combine-rsm-env in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulDelete  47m   instance-set-controller  delete Service kafka-dukzlp-kafka-combine-headless in InstanceSet kafka-dukzlp-kafka-combine successful
  Normal  SuccessfulDelete  47m   instance-set-controller  delete Pod kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful

k3s logs

I0930 03:02:37.766359      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/a1c54c0d-dfa5-47b8-b399-62de6182fac4-scripts\") pod \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\" (UID: \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\") "
I0930 03:02:37.767367      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c69wh\" (UniqueName: \"kubernetes.io/projected/a1c54c0d-dfa5-47b8-b399-62de6182fac4-kube-api-access-c69wh\") pod \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\" (UID: \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\") "
I0930 03:02:37.775634      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c54c0d-dfa5-47b8-b399-62de6182fac4-kube-api-access-c69wh" (OuterVolumeSpecName: "kube-api-access-c69wh") pod "a1c54c0d-dfa5-47b8-b399-62de6182fac4" (UID: "a1c54c0d-dfa5-47b8-b399-62de6182fac4"). InnerVolumeSpecName "kube-api-access-c69wh". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0930 03:02:37.777571      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c54c0d-dfa5-47b8-b399-62de6182fac4-scripts" (OuterVolumeSpecName: "scripts") pod "a1c54c0d-dfa5-47b8-b399-62de6182fac4" (UID: "a1c54c0d-dfa5-47b8-b399-62de6182fac4"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:37.868098      87 reconciler_common.go:289] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/a1c54c0d-dfa5-47b8-b399-62de6182fac4-scripts\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:37.868127      87 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-c69wh\" (UniqueName: \"kubernetes.io/projected/a1c54c0d-dfa5-47b8-b399-62de6182fac4-kube-api-access-c69wh\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:38.097577      87 scope.go:117] "RemoveContainer" containerID="dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"
I0930 03:02:38.110470      87 scope.go:117] "RemoveContainer" containerID="dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"
E0930 03:02:38.111458      87 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f\": not found" containerID="dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"
I0930 03:02:38.111535      87 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"} err="failed to get container status \"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f\": rpc error: code = NotFound desc = an error occurred when try to find container \"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f\": not found"
I0930 03:02:38.904674      87 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c54c0d-dfa5-47b8-b399-62de6182fac4" path="/var/lib/kubelet/pods/a1c54c0d-dfa5-47b8-b399-62de6182fac4/volumes"
I0930 03:02:40.490442      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.490721      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-scripts\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.491017      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.491066      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"tools\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-tools\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.491098      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"jmx-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-jmx-config\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.492822      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kafka-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-kafka-config\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.492980      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t8r2\" (UniqueName: \"kubernetes.io/projected/33229d89-4052-46f1-9b49-ff39382a4c54-kube-api-access-6t8r2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.494079      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-jmx-config" (OuterVolumeSpecName: "jmx-config") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "jmx-config". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:40.495878      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-scripts" (OuterVolumeSpecName: "scripts") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:40.499036      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-kafka-config" (OuterVolumeSpecName: "kafka-config") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "kafka-config". PluginName "kubernetes.io/configmap", VolumeGidValue ""
E0930 03:02:40.500476      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:41.000438084 +0000 UTC m=+4670.562037248 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:40.500641      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:41.0005735 +0000 UTC m=+4670.562172540 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:40.500786      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33229d89-4052-46f1-9b49-ff39382a4c54-kube-api-access-6t8r2" (OuterVolumeSpecName: "kube-api-access-6t8r2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "kube-api-access-6t8r2". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0930 03:02:40.503795      87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-tools" (OuterVolumeSpecName: "tools") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "tools". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:40.594361      87 reconciler_common.go:289] "Volume detached for volume \"tools\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-tools\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594393      87 reconciler_common.go:289] "Volume detached for volume \"jmx-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-jmx-config\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594400      87 reconciler_common.go:289] "Volume detached for volume \"kafka-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-kafka-config\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594425      87 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-6t8r2\" (UniqueName: \"kubernetes.io/projected/33229d89-4052-46f1-9b49-ff39382a4c54-kube-api-access-6t8r2\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594450      87 reconciler_common.go:289] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-scripts\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:41.003851      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:41.004137      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:41.008360      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:42.008326459 +0000 UTC m=+4671.569925498 (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:41.008560      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:42.008534417 +0000 UTC m=+4671.570133498 (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:41.118935      87 scope.go:117] "RemoveContainer" containerID="275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"
I0930 03:02:41.131600      87 scope.go:117] "RemoveContainer" containerID="1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"
I0930 03:02:41.142411      87 scope.go:117] "RemoveContainer" containerID="275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"
E0930 03:02:41.142974      87 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4\": not found" containerID="275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"
I0930 03:02:41.143089      87 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"} err="failed to get container status \"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4\": rpc error: code = NotFound desc = an error occurred when try to find container \"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4\": not found"
I0930 03:02:41.143289      87 scope.go:117] "RemoveContainer" containerID="1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"
E0930 03:02:41.144119      87 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8\": not found" containerID="1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"
I0930 03:02:41.144273      87 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"} err="failed to get container status \"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8\": rpc error: code = NotFound desc = an error occurred when try to find container \"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8\": not found"
I0930 03:02:42.012815      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:42.013091      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:42.017587      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:44.017545543 +0000 UTC m=+4673.579144499 (durationBeforeRetry 2s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:42.018093      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:44.018078084 +0000 UTC m=+4673.579677082 (durationBeforeRetry 2s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:44.032389      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:44.032512      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:44.034265      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:48.034252377 +0000 UTC m=+4677.595851458 (durationBeforeRetry 4s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:44.034879      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:48.034855169 +0000 UTC m=+4677.596454250 (durationBeforeRetry 4s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:48.071111      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:48.071309      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:48.073392      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:56.073379546 +0000 UTC m=+4685.634978502 (durationBeforeRetry 8s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:48.073464      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:56.073454296 +0000 UTC m=+4685.635053293 (durationBeforeRetry 8s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:56.146586      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:56.146823      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:56.148503      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:12.148481508 +0000 UTC m=+4701.710080464 (durationBeforeRetry 16s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:56.148622      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:12.148611133 +0000 UTC m=+4701.710210131 (durationBeforeRetry 16s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:58.273902      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2\") pod \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\" (UID: \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\") "
I0930 03:02:58.273975      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2\") pod \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\" (UID: \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\") "
E0930 03:02:58.276804      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2 podName:0745d15a-c04f-490f-b3e1-ef9e73d9eeac nodeName:}" failed. No retries permitted until 2024-09-30 03:05:00.276780509 +0000 UTC m=+4809.838379465 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2") pod "0745d15a-c04f-490f-b3e1-ef9e73d9eeac" (UID: "0745d15a-c04f-490f-b3e1-ef9e73d9eeac") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:58.276854      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2 podName:0745d15a-c04f-490f-b3e1-ef9e73d9eeac nodeName:}" failed. No retries permitted until 2024-09-30 03:05:00.276835426 +0000 UTC m=+4809.838434423 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2") pod "0745d15a-c04f-490f-b3e1-ef9e73d9eeac" (UID: "0745d15a-c04f-490f-b3e1-ef9e73d9eeac") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:03:12.210663      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:03:12.210849      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:03:12.218308      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:44.218026668 +0000 UTC m=+4733.779737930 (durationBeforeRetry 32s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:03:12.218400      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:44.218380418 +0000 UTC m=+4733.780091638 (durationBeforeRetry 32s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:03:44.277829      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:03:44.277996      87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:03:44.280565      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:04:48.280532377 +0000 UTC m=+4797.843572944 (durationBeforeRetry 1m4s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:03:44.280629      87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:04:48.280621419 +0000 UTC m=+4797.843662028 (durationBeforeRetry 1m4s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Sep 30, 2024
@JashBook JashBook added this to the Release 0.9.1 milestone Sep 30, 2024
@shanshanying
Copy link
Contributor

PVC finalizers are removed and updated. That's why PVCs are deleted before PODs.

@shanshanying shanshanying removed their assignment Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@free6om free6om

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.9.2
Development

No branches or pull requests
4 participants
@Y-Rookie @free6om @JashBook @shanshanying