Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail get DataUploadResult for restore - multiple DataUpload result cms found #7057

Open
zanderson-aim opened this issue Nov 5, 2023 · 28 comments

Comments

@zanderson-aim
Copy link

What steps did you take and what happened:
When trying to restore a backup I get the following error:

    rc-aimit-io:  error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-202311
05205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937
-7654feb4062e,velero.io/resource-usage=DataUpload
                  error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-202311
05205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937
-7654feb4062e,velero.io/resource-usage=DataUpload

The restore command
velero restore create --from-backup velero-imdc-20231105205455 --include-namespaces rc-aimit-io --include-resources persistentvolumeclaims,persistentvolumes,dataupload

What did you expect to happen:
The PV/PVCs would be restored.

The following information will help us better understand what's going on:
bundle-2023-11-05-23-09-42.tar.gz

Anything else you would like to add:
I was able to confirm the data is there by manually mounting the bucket using Kopia tools and restoring the files from there.

Environment:

  • Velero version (use velero version):
Client:
        Version: v1.12.1
        Git commit: 5c4fdfe147357ec7b908339f4516cd96d6b97c61
Server:
        Version: v1.12.1
  • Velero features (use velero client config get features):
    features: <NOT SET>
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17+rke2r1", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T21:42:35Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17+rke2r1", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T21:42:35Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes installer & version:
  • RKE2
  • v1.22.17+rke2r1
@Lyndon-Li
Copy link
Contributor

@zanderson-aim
For a quick check, could you run below command and share the result?
kubectl get cm -n velero

@blackpiglet
Copy link
Contributor

{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}

Please also check the PVCs rc-aimit-io/datadir-rocketchat-mongodb-0 and rc-aimit-io/datadir-rocketchat-mongodb-1 are mounted multiple times.

@zanderson-aim
Copy link
Author

@zanderson-aim For a quick check, could you run below command and share the result? kubectl get cm -n velero

This results in about 9000 lines which look like this

velero-imdc-20231031000006-55gfn-j4jk6   1      5d5h
velero-imdc-20231031000006-55gfn-ldm8d   1      5d5h
velero-imdc-20231031000006-55gfn-qf8hn   1      4d4h
velero-imdc-20231031000006-55gfn-qzkb4   1      4d6h
velero-imdc-20231031000006-7n9b9-c5ph9   1      4d4h
velero-imdc-20231031000006-7n9b9-d8w6b   1      5d5h
velero-imdc-20231031000006-7n9b9-j4gpj   1      4d6h
velero-imdc-20231031000006-7n9b9-ns4xs   1      5d5h
velero-imdc-20231031000006-7wg25-bmgxg   1      4d6h
velero-imdc-20231031000006-7wg25-rml9s   1      4d4h
velero-imdc-20231031000006-7wg25-sd85w   1      5d5h
velero-imdc-20231031000006-7wg25-zm8zq   1      5d5h
velero-imdc-20231031000006-92sqt-5dfmz   1      4d4h
velero-imdc-20231031000006-92sqt-7lhk9   1      4d6h
velero-imdc-20231031000006-92sqt-bdr7h   1      5d5h
velero-imdc-20231031000006-92sqt-kmwf9   1      5d5h
velero-imdc-20231031000006-cdqft-5smd7   1      4d4h
velero-imdc-20231031000006-cdqft-hc99l   1      5d5h
velero-imdc-20231031000006-cdqft-m2lbp   1      5d5h
velero-imdc-20231031000006-cdqft-vwmc7   1      4d6h
velero-imdc-20231031000006-cvmt7-8qx8p   1      5d5h
velero-imdc-20231031000006-cvmt7-gpr97   1      4d6h
velero-imdc-20231031000006-cvmt7-ql5zw   1      4d4h
velero-imdc-20231031000006-cvmt7-rlswx   1      5d5h
velero-imdc-20231031000006-d8zbx-5vfnz   1      4d6h
velero-imdc-20231031000006-d8zbx-cl68t   1      4d4h
velero-imdc-20231031000006-d8zbx-jwgwp   1      5d5h
velero-imdc-20231031000006-d8zbx-t9thk   1      5d5h
velero-imdc-20231031000006-dkzqm-4brdn   1      5d5h
velero-imdc-20231031000006-dkzqm-dpls4   1      4d6h
velero-imdc-20231031000006-dkzqm-vsjfd   1      4d4h
velero-imdc-20231031000006-dkzqm-wf2fr   1      5d5h
velero-imdc-20231031000006-dqqrj-pd4bx   1      5d5h
velero-imdc-20231031000006-dqqrj-pf58j   1      4d4h
velero-imdc-20231031000006-dqqrj-spw8q   1      4d6h

@zanderson-aim
Copy link
Author

{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}

Please also check the PVCs rc-aimit-io/datadir-rocketchat-mongodb-0 and rc-aimit-io/datadir-rocketchat-mongodb-1 are mounted multiple times.

I deleted them from the live system to test the restore, so they don't currently exist any more.

@Lyndon-Li
Copy link
Contributor

@zanderson-aim Are there running data mover restores at the time you run kubectl get cm -n velero?

@zanderson-aim
Copy link
Author

@zanderson-aim Are there running data mover restores at the time you run kubectl get cm -n velero?

No currently nothing running

kubectl -n velero get pods
NAME                      READY   STATUS    RESTARTS      AGE
node-agent-24ct4          1/1     Running   0             11d
node-agent-6ckx4          1/1     Running   0             11d
node-agent-8f559          1/1     Running   0             11d
node-agent-b2kpg          1/1     Running   0             11d
node-agent-pzbnq          1/1     Running   0             11d
node-agent-qzlsh          1/1     Running   0             11d
velero-6798d896db-5hpnk   1/1     Running   1 (40h ago)   11d
# kubectl get cm -n velero | wc -l
9756

@Lyndon-Li
Copy link
Contributor

Lyndon-Li commented Nov 6, 2023

@zanderson-aim
Could you share the content of one of these configMaps? E.g., kubectl get cm -n velero -o yaml velero-imdc-20231031000006-dqqrj-spw8q
Please also let us know the timestamp of the latest one of these configMaps? You can also run kubectl get cm -n velero -o yaml for the latest configMap

@Lyndon-Li
Copy link
Contributor

@zanderson-aim
We want to do several checks on the list of configMaps as you shared, different info are required. So how about this:

  1. Run kubectl get cm -n velero and share us the entire output
  2. Run kubectl get cm -n velero -o yaml and share us the entire output

2 may be very large, not sure if you can make it, but let's see it.

@zanderson-aim
Copy link
Author

zanderson-aim commented Nov 6, 2023

@zanderson-aim We want to do several checks on the list of configMaps as you shared, different info are required. So how about this:

  1. Run kubectl get cm -n velero and share us the entire output
  2. Run kubectl get cm -n velero -o yaml and share us the entire output

2 may be very large, not sure if you can make it, but let's see it.

kubectl get cm -n velero > /tmp/velero-cm-list.txt
velero-cm-list.txt

kubectl get cm -n velero -o yaml > /tmp/velero-cm.yaml
velero-cm.zip

@Lyndon-Li
Copy link
Contributor

Lyndon-Li commented Nov 6, 2023

@zanderson-aim
Looks like you have included velero namespace in the backup as what we can see in the backup describe:

velero.io/v1/Backup:
    - velero/velero-imdc-20231007180046
    - velero/velero-imdc-20231008180047
    - velero/velero-imdc-20231013060021
    - velero/velero-imdc-20231013180021
    - velero/velero-imdc-20231014120022
    - velero/velero-imdc-20231014180023
    - velero/velero-imdc-20231015000059
    - velero/velero-imdc-20231015060059
    - velero/velero-imdc-20231015120059
    - velero/velero-imdc-20231015180000
    - velero/velero-imdc-20231016000000
    - velero/velero-imdc-20231016060000
    - velero/velero-imdc-20231016120001
    - velero/velero-imdc-20231016180001
    - velero/velero-imdc-20231017000001
    - velero/velero-imdc-20231017120047
    - velero/velero-imdc-20231017180047
    - velero/velero-imdc-20231018000047
    - velero/velero-imdc-20231018060048
    - velero/velero-imdc-20231019000049
    - velero/velero-imdc-20231019060049
    - velero/velero-imdc-20231019120049
    - velero/velero-imdc-20231020000032
    - velero/velero-imdc-20231020060038
    - velero/velero-imdc-20231020120038
    - velero/velero-imdc-20231020180039
    - velero/velero-imdc-20231021000039
    - velero/velero-imdc-20231021060013
    - velero/velero-imdc-20231021120013
    - velero/velero-imdc-20231021180011
    - velero/velero-imdc-20231022000012
    - velero/velero-imdc-20231022060012
    - velero/velero-imdc-20231022120053
    - velero/velero-imdc-20231022180054
    - velero/velero-imdc-20231023000054
    - velero/velero-imdc-20231023060054
    - velero/velero-imdc-20231023120057
    - velero/velero-imdc-20231023180057
    - velero/velero-imdc-20231024000057
    - velero/velero-imdc-20231024060058
    - velero/velero-imdc-20231024120058
    - velero/velero-imdc-20231024180058
    - velero/velero-imdc-20231025000059
    - velero/velero-imdc-20231025060059
    - velero/velero-imdc-20231025120049
    - velero/velero-imdc-20231025180049
    - velero/velero-imdc-20231026000059
    - velero/velero-imdc-20231026060059
    - velero/velero-imdc-20231026120000
    - velero/velero-imdc-20231026180000
    - velero/velero-imdc-20231027000001
    - velero/velero-imdc-20231027060001
    - velero/velero-imdc-20231027120001
    - velero/velero-imdc-20231027180001
    - velero/velero-imdc-20231028000002
    - velero/velero-imdc-20231028060002
    - velero/velero-imdc-20231028120002
    - velero/velero-imdc-20231028180003
    - velero/velero-imdc-20231029000003
    - velero/velero-imdc-20231029060003
    - velero/velero-imdc-20231029120004
    - velero/velero-imdc-20231029180004
    - velero/velero-imdc-20231030000004
    - velero/velero-imdc-20231030060005
    - velero/velero-imdc-20231030120005
    - velero/velero-imdc-20231030180005
    - velero/velero-imdc-20231030183306
    - velero/velero-imdc-20231031000006
    - velero/velero-imdc-20231031060006
    - velero/velero-imdc-20231031120006
    - velero/velero-imdc-20231031180007
    - velero/velero-imdc-20231101000007
    - velero/velero-imdc-20231101060008
    - velero/velero-imdc-20231101120008
    - velero/velero-imdc-20231101180008
    - velero/velero-imdc-20231101201654
    - velero/velero-imdc-20231102000009
    - velero/velero-imdc-20231102060009
    - velero/velero-imdc-20231102120009
    - velero/velero-imdc-20231102180010
    - velero/velero-imdc-20231103000010
    - velero/velero-imdc-20231103060011
    - velero/velero-imdc-20231103120011
    - velero/velero-imdc-20231103180011
    - velero/velero-imdc-20231104000012
    - velero/velero-imdc-20231104060012
    - velero/velero-imdc-20231104120045
    - velero/velero-imdc-20231104180045
    - velero/velero-imdc-20231105000046
    - velero/velero-imdc-20231105060046
    - velero/velero-imdc-20231105120046
    - velero/velero-imdc-20231105180046
    - velero/velero-imdc-20231105205455
  velero.io/v1/BackupRepository:
    - velero/4luvofthegame-com-backblaze-imdc-velero-kopia-lgzfh
    - velero/4x5program-org-backblaze-imdc-velero-kopia-xgzg5
    - velero/alumcoprecision-com-backblaze-imdc-velero-kopia-8s4wv
    - velero/argocd-backblaze-imdc-velero-kopia-lmbz8
    - velero/blingtour-com-backblaze-imdc-velero-kopia-n9s8j
    - velero/budibase-backblaze-imdc-velero-kopia-wnccx
    - velero/camplulu-com-backblaze-imdc-velero-kopia-zt6b5
    - velero/ceph-csi-backblaze-imdc-velero-kopia-675cp
    - velero/drillrdd-com-backblaze-imdc-velero-kopia-5j7kz
    - velero/fleet-aimit-io-backblaze-imdc-velero-kopia-kht45
    - velero/grafana-agent-backblaze-imdc-velero-kopia-kvmx2
    - velero/hs-aimit-io-backblaze-imdc-velero-kopia-576hs
    - velero/ingress-contour-backblaze-imdc-velero-kopia-v85kn
    - velero/inventory-aimit-io-backblaze-imdc-velero-kopia-2tl79
    - velero/kube-system-backblaze-imdc-velero-kopia-xc97l
    - velero/pm-aimit-io-backblaze-imdc-velero-kopia-dk9q8
    - velero/raisingthebar-org-backblaze-imdc-velero-kopia-4kfpb
    - velero/rc-aimit-io-backblaze-imdc-velero-kopia-gpvx9
    - velero/test-velero-backblaze-imdc-velero-kopia-8p58x
    - velero/velero-backblaze-imdc-velero-kopia-gsnhb
    - velero/velero-test-backblaze-imdc-velero-kopia-d5z69
  velero.io/v1/BackupStorageLocation:
    - velero/backblaze-imdc-velero
  velero.io/v1/Restore:
    - velero/velero-imdc-20231030183306-20231030204827
    - velero/velero-imdc-20231030183306-20231031223223
    - velero/velero-imdc-20231030183306-20231031225141
    - velero/velero-imdc-20231030183306-20231031225242
    - velero/velero-imdc-20231030183306-20231031231546
    - velero/velero-imdc-20231030183306-20231031231633
    - velero/velero-imdc-20231030183306-20231031231704
    - velero/velero-imdc-20231030183306-20231101025845
    - velero/velero-imdc-20231030183306-20231101032154
    - velero/velero-imdc-20231031120006-20231031221148
    - velero/velero-imdc-20231031180007-20231031214800
    - velero/velero-imdc-20231101201654-20231101205439
    - velero/velero-imdc-20231101201654-20231101225032
  velero.io/v1/Schedule:
    - velero/velero-imdc
  velero.io/v2alpha1/DataDownload:
    - velero/velero-imdc-20231030183306-20231030204827-hcrbx
    - velero/velero-imdc-20231031180007-20231031214800-cssf6
    - velero/velero-imdc-20231101201654-20231101205439-hvhjs
    - velero/velero-imdc-20231101201654-20231101205439-m7fpx
    - velero/velero-imdc-20231101201654-20231101205439-tvrnb
  velero.io/v2alpha1/DataUpload:
    - velero/velero-imdc-20231014120022-22nb7
    - velero/velero-imdc-20231014120022-26cqh
    - velero/velero-imdc-20231014120022-2jj5l
    - velero/velero-imdc-20231014120022-2wtwh
    - velero/velero-imdc-20231014120022-2zrlt
    - velero/velero-imdc-20231014120022-428xg
    - velero/velero-imdc-20231014120022-5wztf
    - velero/velero-imdc-20231014120022-62sb9
    - velero/velero-imdc-20231014120022-77k7p
    - velero/velero-imdc-20231014120022-7fzjj
    - velero/velero-imdc-20231014120022-8mqzs
    - velero/velero-imdc-20231014120022-9jdtp
    - velero/velero-imdc-20231014120022-b8gm4
    - velero/velero-imdc-20231014120022-bcsvm
    - velero/velero-imdc-20231014120022-d275t
    - velero/velero-imdc-20231014120022-d5dv7
    - velero/velero-imdc-20231014120022-d8xc7
    - velero/velero-imdc-20231014120022-df8h8
    - velero/velero-imdc-20231014120022-drxv4
    - velero/velero-imdc-20231014120022-dtj4z
    - velero/velero-imdc-20231014120022-dwhv9
    - velero/velero-imdc-20231014120022-hfmlh
    - velero/velero-imdc-20231014120022-hj9b8
    - velero/velero-imdc-20231014120022-jgdj5
    - velero/velero-imdc-20231014120022-k57c9
    - velero/velero-imdc-20231014120022-kg59k
    - velero/velero-imdc-20231014120022-lgcxg
    - velero/velero-imdc-20231014120022-ls8s5
    - velero/velero-imdc-20231014120022-m4wmc
    - velero/velero-imdc-20231014120022-mrksz
    - velero/velero-imdc-20231014120022-n6nbs
    - velero/velero-imdc-20231014120022-p6hmd
    - velero/velero-imdc-20231014120022-pcbpj
    - velero/velero-imdc-20231014120022-qkxpf

Most resources in the velero namespace cannot be backed up, if you want to backup the entire cluster, please exclude velero namespace.

@zanderson-aim
Copy link
Author

Makes sense, I will update the job and test again. I guess these makes all the "old" backups that include velero already unrestorable?

@Lyndon-Li
Copy link
Contributor

I guess these makes all the "old" backups that include velero already unrestorable?

yes

@shawn-hurley
Copy link
Contributor

If this is true, can we change the default to permanently exclude the velero namespace, or even better, maybe, always exclude velero resources by default?

@zanderson-aim
Copy link
Author

I can confirm it worked after I excluded the namespace.

@zanderson-aim
Copy link
Author

I did tried some older backups before 1.12 and they worked fine with the namespace included. Must be something with changes in 1.12.

@Lyndon-Li Lyndon-Li self-assigned this Nov 9, 2023
@Lyndon-Li
Copy link
Contributor

I did tried some older backups before 1.12 and they worked fine with the namespace included. Must be something with changes in 1.12.

OK, I will keep this issue open and see if we can make some improvement around DataUpload CRs.
However, we will not take this as high priority since we don't see too much value to include velero namespace into the backup, and even it is required, the DataUpload CRs could be filtered out as a workaround.

@shawn-hurley
Copy link
Contributor

@shubham-pampattiwar @sseago We probably want to make sure that this is documented in our downstream docs

@oboudry-mvp
Copy link

For info I had exactly the same issue with v.12.1, restore of snapshots (Kopia to Minio) were failing when velero was included in the backup. I excluded all kube- namespaces but left velero in.

New backups without velero namespace restore fine.

Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

@filipe-silva-magalhaes-alb

I have the same issue.

I solved this by cleaning the s3 bucket, installing the Velero doing the backup again, and restoring.

Kopia might get confused when the bucket has multiple backups.

Copy link

github-actions bot commented Aug 4, 2024

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

@github-actions github-actions bot added the staled label Aug 4, 2024
@Lyndon-Li Lyndon-Li removed the staled label Aug 5, 2024
@dharanui
Copy link

dharanui commented Oct 8, 2024

Facing same issue when trying to restore with velero version 1.14

@Norris1z
Copy link

I faced the same issue with v1.15.0 and here is how I manged to get my PVC data from the backup.

For context: I deleted my postgres namespace and wanted to restore it with velero restore create --from-backup 2024-11-09-1 --include-namespaces postgres but got the error

Fail to restore from DataUploadResult: fail get DataUploadResult for restore: 2024-11-09-1-20241110161813: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=postgres.postgres-volume-claim,velero.io/restore-uid=b9cb9487-31ee-4434-93f1-40fd076e13d6,velero.io/resource-usage=DataUpload

I found this issue after googling and panicked after seeing #7057 (comment)

Thanks to https://cloudcasa.io/blog/how-to-restore-velero-backup-data-without-velero/ this is how I fixed my issue:

  • First installed Kopia from https://kopia.io/docs/installation/
  • Then connect to the backup repository
    kopia repository connect s3 --bucket=velero --prefix=kopia/postgres --endpoint="yourendpoint" --access-key=xxxx --secret-access-key=xxxx --region=us-east-1 --disable-tls
  • this would prompt for a password which can then be found at
    encryptionKey = "static-passw0rd"
  • my bucket is velero and has a kopia folder in there with a postgres directory in it.
  • run kopia snapshot list --all -l to get all the snapshots for my case was
default@default:snapshot-data-upload-download/kopia/postgres/postgres-volume-claim
  2024-11-09 04:23:36 CET ke7c2317576cfc5336a966ee040d68a3d 493.8 MB drwxr-xr-x files:5136 dirs:35 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1) pins:velero-pin
  • Then run kopia snapshot restore ke7c2317576cfc5336a966ee040d68a3d /path-to-restore-to.
  • I then recreated the postgres PVC and found the path it was mounted on. (check which node the pod is created on and find the PVC name with the find command).
  • Then I copied the restore to that path.

This fixed the issue for me. I hope this helps someone too!

@roman-parkhunovskyi
Copy link

roman-parkhunovskyi commented Nov 29, 2024

Had the same Fail get DataUploadResult for restore - multiple DataUpload result cms found restoring from the scheduled full backup (velero 1.14.1). Thanks to sighupio/fury-distribution#304 (comment), tried with a manual backup restore and had no issue.
This is the most annoying bug so far in velero.

@ppapp92
Copy link

ppapp92 commented Dec 19, 2024

Also facing this issue with a scheduled backups. Performing a manual backup and restoring from it works perfectly fine. This seems like a major bug is scheduled backups are not usable.

Issue happens on 1.14.0, 1.14.1, 1.15.0

@adityagu0910
Copy link

Also facing the same issue in restore with csi snapshot data mover
Errors: Velero: <none> Cluster: <none> Namespaces: nginx-example: error preparing persistentvolumeclaims/nginx-example/nginx-logs: rpc error: code = Unknown desc = fail get DataUploadResult for restore: restore-nginx-example: multip le DataUpload result cms found with labels velero.io/pvc-namespace-name=nginx-example.nginx-logs,velero.io/restore-uid=b19c0d7d-0a94-4c08-a953-8151de2966f8,velero.io/resource-usage=DataU pload

@Siradjedd
Copy link

i have the same issue when working with CSI and datamover. i tried to backup a namespace in clusterA and restore it in clusterB using velero and longhorn. the backup worked. but when i try to restore on the clusterB i got these issues:
velero-describe:

`Name:         labadeiz-longhorn-backup-datamover-20250107110959
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       PartiallyFailed (run 'velero restore logs labadeiz-longhorn-backup-datamover-20250107110959' for more information)
Total items to be restored:  303
Items restored:              303

Started:    2025-01-07 11:09:59 +0100 CET
Completed:  2025-01-07 11:10:22 +0100 CET

Warnings:
  Velero:     <none>
  Cluster:  could not restore, CustomResourceDefinition "apps.catalog.cattle.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "backups.postgresql.cnpg.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "certificates.cert-manager.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "clusters.postgresql.cnpg.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "opensearchclusters.opensearch.opster.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "scheduledbackups.postgresql.cnpg.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "servicemonitors.monitoring.coreos.com" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, ClusterRole "system:auth-delegator" already exists. Warning: the in-cluster version is different than the backed-up version
  Namespaces:
    labadeiz:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "cluster-pg-release" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "cnpg-operator-release" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "filebeat-release" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "minio-operator" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "minio-tenant" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "minio-vault" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "nginxservice" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "oper-install" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, App "os" already exists. Warning: the in-cluster version is different than the backed-up version
               could not restore, Lease "minio-operator-lock" already exists. Warning: the in-cluster version is different than the backed-up version

Errors:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    labadeiz:  error preparing persistentvolumeclaims/labadeiz/cluster-pg-release-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.cluster-pg-release-1,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/cluster-pg-release-2: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.cluster-pg-release-2,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/cluster-pg-release-3: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.cluster-pg-release-3,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/data-0-adeizkfk-kafka-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.data-0-adeizkfk-kafka-0,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/data-adeiz-opensearch-cluster-nodes-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.data-adeiz-opensearch-cluster-nodes-0,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/data-adeiz-opensearch-cluster-nodes-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.data-adeiz-opensearch-cluster-nodes-1,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/data-adeizkfk-zookeeper-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.data-adeizkfk-zookeeper-0,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/data0-myminio-pool-0-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.data0-myminio-pool-0-0,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error preparing persistentvolumeclaims/labadeiz/nfspvc: rpc error: code = Unknown desc = fail get DataUploadResult for restore: labadeiz-longhorn-backup-datamover-20250107110959: no DataUpload result cm found with labels velero.io/pvc-namespace-name=labadeiz.nfspvc,velero.io/restore-uid=ab0fb103-356d-4e90-b561-d74e5c030c36,velero.io/resource-usage=DataUpload
               error restoring opensearchclusters.opensearch.opster.io/labadeiz/adeiz-opensearch-cluster: create not allowed while custom resource definition is terminating

Backup:  labadeiz-longhorn-backup-datamover

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  auto

CSI Snapshot Restores: <none included>

Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s

Preserve Service NodePorts:  auto

Uploader config:


HooksAttempted:   0
HooksFailed:      0
`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests