Deleting PVCs created by VolumeClaimTemplate is flaky

[QuanZhang-William]

Expected Behavior

pvcs created by VolumeClaimTemplate should be deleted when the owning pipelinerun is completed with coschedule:pipelineruns.

Actual Behavior

There is race condition, some pvcs are randomly left in terminating status

Steps to Reproduce the Problem

1. Set disable-affinity-assistant: "true" and coschedule: "pipelineruns"
2. Run the example PipelineRun multiple times

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: demo-pipeline-run-template-only-
spec:
  pipelineSpec:
    workspaces:
    - name: workspace1
    - name: workspace2
    tasks:
    - name: foo
      workspaces:
      - name: workspace1
      - name: workspace2
      taskSpec:
        steps:
        - image: busybox
          script: echo "$(workspaces.workspace1.path) $(workspaces.workspace2.path)"
  workspaces:
  - name: workspace1
    volumeClaimTemplate:
      metadata:
        name: workspace1
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 16Mi
  - name: workspace2
    volumeClaimTemplate:
      metadata:
        name: workspace2
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 16Mi

3. Check if pvcs and some PVCs are left in terminating status randomly

$ k get pvc
NAME                                                    STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
workspace1-4ed6374fea-affinity-assistant-c37495a2b7-0   Terminating   pvc-fa53b790-8811-403e-a6a2-dabe9e817d55   1Gi        RWO            standard       2m29s

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.8", GitCommit:"0ce7342c984110dfc93657d64df5dc3b2c0d1fe9", GitTreeState:"clean", BuildDate:"2023-03-15T13:39:54Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.10-gke.2700", GitCommit:"df2448d7739c44e41ee999b9065ce506cab86344", GitTreeState:"clean", BuildDate:"2023-06-22T09:25:37Z", GoVersion:"go1.19.9 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.51.0

[QuanZhang-William]

Thanks for reporting the bug @jimmyjones2 (see #5776 (comment))!

The race condition is caused by the wrong order of purging finalizer and deleting PVCs.

The root cause is that the k8s PVCProtectionController adds the finalizer back if it detects the PVC is being used by a pod and the the kubernetes.io/pvc-protection is removed by Tekton. If the add back action happens before the PVC deletion call is completed, the PVC will be left in terminating status.

Switching the the order of the calls should fix the race condition.

/assign