Velero selects wrong snapshotclass for CSI backed PVC's when multiple VolumeSnapshotClass resources available

[togoschi]

We've deployed the v1beta1 snapshot feature on kubernetes and testing Velero's CSI feature for PVC backup's. Basically it works but with multiple SnapshotClasses available Velero doesn't pick up the right snapshotClass for the corresponding StorageClass. It seems that behaviour is random though the comment https://github.com/vmware-tanzu/velero-plugin-for-csi/blob/v0.1.2/internal/util/util.go#L127 indicates that's already implemented. Is it possible that we miss any annotation/parameter to select explicitly the VolumeSnapshotClass for a backup resource? In addition changing the snapshot.storage.kubernetes.io/is-default-class annotation of the volumeSnapshotClass definition has no impact on this malformed behaviour.

Our test environment runs with

• Kubernetes v1.18
• Ceph-CSI v3.2.1
• CSI external-snapshotter v3.0.3
• Velero v1.5.2 (+ v1.5.3) with enabled CSI feature
• Velero Plugin-for-CSI v0.1.2
• multiple deployed RBD StorageClass resources accompanied by its corresponding VolumeSnapshotClass resources

Each VolumeSnapshotClass uses the same provisioner and is illustated by the following example definition:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: csi-rbd-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "false"
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: rbd.csi.ceph.com
deletionPolicy: Delete
parameters:
  clusterID: xxxxxxxx-xxxx-xxxx-xxxxxxxx
  csi.storage.k8s.io/snapshotter-secret-name: csi-rbd-secret
  csi.storage.k8s.io/snapshotter-secret-namespace: kube-system

[dsu-igeek]

Would you add log files please for a backup that is failing.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

[togoschi]

• Velero Controller (Pod) Log

# kubectl logs deployment/velero -n velero
velero-pod.2021-02-02.log

• Description of the backup resource

# velero backup describe testbackup -n backup --details

Name:         testbackup
Namespace:    backup
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.10
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18
Phase:  Completed
Errors:    0
Warnings:  3
Namespaces:
  Included:  default
  Excluded:  <none>
Resources:
  Included:        persistentVolumeClaim
  Excluded:        <none>
  Cluster-scoped:  auto
Label selector:  <none>
Storage Location:  default
Velero-Native Snapshot PVs:  auto
TTL:  720h0m0s
Hooks:  <none>
Backup Format Version:  1.1.0
...
Total items to be backed up:  1
Items backed up:              1
Resource List:
  v1/PersistentVolumeClaim:
    - default/csi-rbd0
Velero-Native Snapshots: <none included>

• Backup resource logs

# velero backup logs testbackup -n backup
velero-testbackup.log

• Output from PVC, VolumeSnapshot and VolumeSnapshotContent

# kubectl describe pvc csi-rbd0

Name:          csi-rbd0
Namespace:     default
StorageClass:  csi-rbd-sc
Status:        Bound
Volume:        pvc-e85c419d-0311-4225-b839-94c8752522cf
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com

# kubectl describe volumesnapshot velero-csi-rbd0-64kgj

Name:         velero-csi-rbd0-64kgj
Namespace:    default
Labels:       velero.io/backup-name=testbackup
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshot
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generate Name:  velero-csi-rbd0-
  Generation:     1
...
Spec:
  Source:
    Persistent Volume Claim Name:  csi-rbd0
  Volume Snapshot Class Name:      csi-rbd-ec-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-55ba3c49-f9a0-49bd-8212-ee41f9b7a317
  Ready To Use:                        false
Events:
  Type    Reason            Age   From                 Message
  ----    ------            ----  ----                 -------
  Normal  CreatingSnapshot  18m   snapshot-controller  Waiting for a snapshot default/velero-csi-rbd0-64kgj to be created by the CSI driver.

# kubectl describe volumesnapshotcontent snapcontent-55ba3c49-f9a0-49bd-8212-ee41f9b7a317

Name:         snapcontent-55ba3c49-f9a0-49bd-8212-ee41f9b7a317
Namespace:    
Labels:       <none>
Annotations:  snapshot.storage.kubernetes.io/deletion-secret-name: csi-rbd-ec-secret
              snapshot.storage.kubernetes.io/deletion-secret-namespace: kube-system
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshotContent
Metadata:
  Creation Timestamp:  2021-02-02T16:49:15Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  Generation:  1
...
Spec:
  Deletion Policy:  Delete
  Driver:           rbd.csi.ceph.com
  Source:
    Volume Handle:             0001-0024-0c930eb1-3e3d-43a4-a746-01ea8c4b58fe-0000000000000008-0389661c-6576-11eb-a097-f60d898ac7a3
  Volume Snapshot Class Name:  csi-rbd-ec-snapclass
  Volume Snapshot Ref:
    API Version:       snapshot.storage.k8s.io/v1beta1
    Kind:              VolumeSnapshot
    Name:              velero-csi-rbd0-64kgj
    Namespace:         default
    Resource Version:  333081087
    UID:               55ba3c49-f9a0-49bd-8212-ee41f9b7a317
Events:                <none>

• Involved VolumeSnapshotClass Resources

# kubectl describe volumesnapshotclass csi-rbd-ec-snapclass

Name:             csi-rbd-ec-snapclass
Namespace:        
Labels:           velero.io/csi-volumesnapshot-class=true
Annotations:      snapshot.storage.kubernetes.io/is-default-class: true
API Version:      snapshot.storage.k8s.io/v1beta1
Deletion Policy:  Delete
Driver:           rbd.csi.ceph.com
Kind:             VolumeSnapshotClass
...
Parameters:
  Cluster ID:                                       0c930eb1-3e3d-43a4-a746-01ea8c4b58fe
  csi.storage.k8s.io/snapshotter-secret-name:       csi-rbd-ec-secret
  csi.storage.k8s.io/snapshotter-secret-namespace:  kube-system

# kubectl describe volumesnapshotclass csi-rbd-snapclass

Name:             csi-rbd-snapclass
Namespace:        
Labels:           velero.io/csi-volumesnapshot-class=true
Annotations:      snapshot.storage.kubernetes.io/is-default-class: false
API Version:      snapshot.storage.k8s.io/v1beta1
Deletion Policy:  Delete
Driver:           rbd.csi.ceph.com
Kind:             VolumeSnapshotClass
...
Parameters:
  Cluster ID:                                       0c930eb1-3e3d-43a4-a746-01ea8c4b58fe
  csi.storage.k8s.io/snapshotter-secret-name:       csi-rbd-secret
  csi.storage.k8s.io/snapshotter-secret-namespace:  kube-system

[togoschi]

Again i want to underline that it doesn't matter if the snapshot.storage.kubernetes.io/is-default-class annotation in the snapclass definition is true or false. And when i delete the "wrong" volumeSnapshotClass - in this case csi-rbd-ec-snapclass - Velero picks up the right class (csi-rbd-snapclass).