Can't perform a restore volume from s3 using kopia to ebs.csi.aws.com provisioner

[ametdoohan]

What steps did you take and what happened:

When doing restore using velero restore create restore-kopia --from-backup default-ns-5dec2023 --restore-volumes --include-namespaces default all resource is restored but pvc data is not restored.

I'm deploying this pod :

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 1Gi

from datadownloads is completed:

apiVersion: velero.io/v2alpha1
kind: DataDownload
metadata:
  creationTimestamp: "2023-12-05T09:20:12Z"
  generateName: restore-kopia-
  generation: 6
  labels:
    velero.io/accepted-by: ip-10-175-164-113.ap-southeast-3.compute.internal
    velero.io/async-operation-id: dd-2e47d2a1-076a-4666-8022-53cb3b042088.f973ed43-a5ab-4ea41ba74
    velero.io/restore-name: restore-kopia
    velero.io/restore-uid: 2e47d2a1-076a-4666-8022-53cb3b042088
  name: restore-kopia-sgnxk
  namespace: velero
  ownerReferences:
  - apiVersion: velero.io/v1
    controller: true
    kind: Restore
    name: restore-kopia
    uid: 2e47d2a1-076a-4666-8022-53cb3b042088
  resourceVersion: "129825408"
  uid: 452203ef-cae5-4bd3-935b-3f0d96147b12
spec:
  backupStorageLocation: default
  operationTimeout: 4h0m0s
  snapshotID: 0b5c652b93c04d04be87572848df9ab1
  sourceNamespace: default
  targetVolume:
    namespace: default
    pv: ""
    pvc: ebs-claim
status:
  completionTimestamp: "2023-12-05T09:20:38Z"
  node: ip-10-175-164-113.ap-southeast-3.compute.internal
  phase: Completed
  progress:
    bytesDone: 1010
    totalBytes: 1010
  startTimestamp: "2023-12-05T09:20:12Z"

What did you expect to happen:

im expecting able to restore from s3 kopia to pvc.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

bundle-2023-12-05-16-39-57.tar.gz

Anything else you would like to add:

Environment:

• Velero version (use velero version): v1.12.1
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): Client Version: v1.28.3, Server Version: v1.26.10-eks-4f4795d
• Kubernetes installer & version: EKS 1.26.10
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Amazon Linux 2

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

What is the file size in the source volume? Is it 1010?

[ametdoohan]

What is the file size in the source volume? Is it 1010?

yes 1010, but when do restore there is no data inside restored PVC.

[Lyndon-Li]

Can you describe the file? What is it used for? Is it possibly manipulated by the workload you are trying to restore?
Furthermore, you can use he kopia mount command to mount the backup repository and check if the file is alright, so that we know whether the problem comes from backup or restore.

[Lyndon-Li]

Some more info:

• From the pod info, looks like you are trying to backup the PVC efs-claim which is a EFS volume. However, EFS doesn't support CSI snapshot, so CSI snapshot data movement doesn't work with it.
• From the attached log, there are indeed two DataUploads, but they are for the volume aws-iam-token, not sure if this is really the volume you want to backup, but since this volume is managed by aws IAM, the files may be changed after the pod is restored, so you cannot expect you see the same thing as the source pod.

[ametdoohan]

Can you describe the file? What is it used for? Is it possibly manipulated by the workload you are trying to restore? Furthermore, you can use he kopia mount command to mount the backup repository and check if the file is alright, so that we know whether the problem comes from backup or restore.

running pod is only for print $(date -u) on /data/out.txt , also I've trying connect kopia from my local using this command but getting this error

kopia repository connect s3 --bucket=[redacted] --access-key=[redacted] --secret-access-key=[redacted] --session-token=[redacted] --region=ap-southeast-3

ERROR error connecting to repository: repository not initialized in the provided storage

Some more info:

• From the pod info, looks like you are trying to backup the PVC efs-claim which is a EFS volume. However, EFS doesn't support CSI snapshot, so CSI snapshot data movement doesn't work with it.

I've been edited, that's the old manifest. I'm using ebs volume. here is my ebs-sc

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"ebs-sc"},"provisioner":"ebs.csi.aws.com","volumeBindingMode":"Immediate"}
  creationTimestamp: "2023-12-01T07:59:55Z"
  name: ebs-sc
  resourceVersion: "128145341"
  uid: c8dd25f2-7739-4f02-98e4-f88ee1fedd93
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

• From the attached log, there are indeed two DataUploads, but they are for the volume aws-iam-token, not sure if this is really the volume you want to backup, but since this volume is managed by aws IAM, the files may be changed after the pod is restored, so you cannot expect you see the same thing as the source pod.

yes, that's volume I want to backup, it just different backup name, but when restore all data is gone.

[Lyndon-Li]

@ametdoohan So it means you wanted to backup the token files in volume aws-iam-token, and after restore, you expect the same files exist in the restored pod?

[Lyndon-Li]

dup with #7189