Restore fails due to Restic timeout #3250

svua · 2020-11-25T20:39:12Z

svua
Nov 25, 2020

How I can restore the backup?

velero version 
Client:
        Version: v1.5.2
        Git commit: e115e5a191b1fdb5d379b62a35916115e77124a4
Server:
        Version: v1.5.2

kubectl version 
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

velero backup create jenkins-test-selector --default-volumes-to-restic --selector app.kubernetes.io/name=jenkins

velero backup get
NAME                           STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
jenkins-test-selector          Completed   0        0          2020-11-25 18:02:29 +0200 EET   29d       default            app.kubernetes.io/name=jenkins

velero backup describe jenkins-test-selector --details

Name:         jenkins-test-selector
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.3
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app.kubernetes.io/name=jenkins

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2020-11-25 18:02:29 +0200 EET
Completed:  2020-11-25 18:07:45 +0200 EET

Expiration:  2020-12-25 18:02:29 +0200 EET

Total items to be backed up:  26
Items backed up:              26

Resource List:
  apps/v1/Deployment:
    - core/jenkins
  apps/v1/ReplicaSet:
    - core/jenkins-554866c75b
    - core/jenkins-5747656d46
    - core/jenkins-57845bdf95
    - core/jenkins-57cd7d4d8
    - core/jenkins-5c6fdd87f8
    - core/jenkins-66ccb89dc7
    - core/jenkins-6dd85ff744
    - core/jenkins-7948dc55d7
    - core/jenkins-7d787bcf9
    - core/jenkins-868647575d
    - core/jenkins-8798b6889
  rbac.authorization.k8s.io/v1/Role:
    - core/jenkins-casc-reload
    - core/jenkins-schedule-agents
  rbac.authorization.k8s.io/v1/RoleBinding:
    - core/jenkins-schedule-agents
    - core/jenkins-watch-configmaps
  v1/ConfigMap:
    - core/jenkins
  v1/Endpoints:
    - core/jenkins
    - core/jenkins-agent
  v1/PersistentVolume:
    - local-pv-ed289748
  v1/PersistentVolumeClaim:
    - core/jenkins
  v1/Pod:
    - core/jenkins-57845bdf95-bd2k6
  v1/Secret:
    - core/jenkins
  v1/Service:
    - core/jenkins
    - core/jenkins-agent
  v1/ServiceAccount:
    - core/jenkins

Velero-Native Snapshots: <none included>

Restic Backups:
  Completed:
    core/jenkins-57845bdf95-bd2k6: jenkins-home, plugin-dir, plugins, secrets-dir, stash-podinfo, tmp, tmp-dir

velero restore create jenkins-test-selector-restore --from-backup jenkins-test-selector --namespace-mappings core:jenkins-restore
Restore request "jenkins-test-selector-restore" submitted successfully.
Run `velero restore describe jenkins-test-selector-restore` or `velero restore logs jenkins-test-selector-restore` for more details.

velero restore logs jenkins-test-selector-restore
Logs for restore "jenkins-test-selector-restore" are not available until it's finished processing. Please wait until the restore has a phase of Completed or Failed and try again.

velero restore get
NAME                            BACKUP                  STATUS   STARTED   COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
jenkins-test-selector-restore   jenkins-test-selector   New      <nil>     <nil>       0        0          2020-11-25 22:27:02 +0200 EET   <none>


velero restore describe jenkins-test-selector-restore
Name:         jenkins-test-selector-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  

Started:    <n/a>
Completed:  <n/a>

Backup:  jenkins-test-selector

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Namespace mappings:  core=jenkins-restore

Label selector:  <none>

Restore PVs:  auto

kubectl logs -f deployment/velero -n velero
time="2020-11-25T20:27:09Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2020-11-25T20:27:09Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2020-11-25T20:27:09Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
time="2020-11-25T20:28:09Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2020-11-25T20:28:09Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2020-11-25T20:28:09Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"

zubron · 2020-11-26T13:14:58Z

zubron
Nov 26, 2020

Processing a new restore may not happen instantaneously but I would have expected some entries in the velero deployment log from the restore controller. Could you provide us with the full logs from the velero deployment using
kubectl logs deployment/velero -n velero, without the follow flag? You can paste long output into a GitHub gist if preferred.

0 replies

svua · 2020-11-27T08:19:21Z

svua
Nov 27, 2020
Author

I want to restore a backup in the same cluster in a different namespace

New attempt to deploy backup

k create ns jenkins-restore
namespace/jenkins-restore created


velero get backup
NAME                           STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
jenkins-daily-20201124000052   Completed   0        0          2020-11-24 02:00:52 +0200 EET   26d       default            <none>
jenkins-daily-20201123000052   Completed   0        0          2020-11-23 02:00:52 +0200 EET   25d       default            <none>
jenkins-daily-20201122000052   Completed   0        0          2020-11-22 02:00:52 +0200 EET   24d       default            <none>
jenkins-daily-20201121000052   Completed   0        0          2020-11-21 02:00:52 +0200 EET   23d       default            <none>
jenkins-daily-20201120000052   Completed   0        0          2020-11-20 02:00:52 +0200 EET   22d       default            <none>
jenkins-daily-20201119000051   Completed   0        0          2020-11-19 02:00:51 +0200 EET   21d       default            <none>
jenkins-daily-20201118000051   Completed   0        0          2020-11-18 02:00:51 +0200 EET   20d       default            <none>
jenkins-daily-20201117000051   Completed   0        0          2020-11-17 02:00:51 +0200 EET   19d       default            <none>
jenkins-daily-20201116000051   Completed   0        0          2020-11-16 02:00:51 +0200 EET   18d       default            <none>
jenkins-daily-20201115000051   Completed   0        0          2020-11-15 02:00:51 +0200 EET   17d       default            <none>
jenkins-daily-20201114000051   Completed   0        0          2020-11-14 02:00:51 +0200 EET   16d       default            <none>
jenkins-daily-20201113000050   Completed   0        0          2020-11-13 02:00:50 +0200 EET   15d       default            <none>
jenkins-daily-20201112000050   Completed   0        0          2020-11-12 02:00:50 +0200 EET   14d       default            <none>
jenkins-daily-20201111000050   Completed   0        0          2020-11-11 02:00:50 +0200 EET   13d       default            <none>
jenkins-daily-20201110000050   Completed   0        0          2020-11-10 02:00:50 +0200 EET   12d       default            <none>
jenkins-daily-20201109000050   Completed   0        0          2020-11-09 02:00:50 +0200 EET   11d       default            <none>
jenkins-daily-20201108000049   Completed   0        0          2020-11-08 02:00:49 +0200 EET   10d       default            <none>
jenkins-daily-20201107000049   Completed   0        0          2020-11-07 02:00:49 +0200 EET   9d        default            <none>
jenkins-daily-20201106000049   Completed   0        0          2020-11-06 02:00:49 +0200 EET   8d        default            <none>
jenkins-daily-20201105000049   Completed   0        0          2020-11-05 02:00:49 +0200 EET   7d        default            <none>
jenkins-daily-20201104000049   Completed   0        0          2020-11-04 02:00:49 +0200 EET   6d        default            <none>
jenkins-daily-20201103000048   Completed   0        0          2020-11-03 02:00:48 +0200 EET   5d        default            <none>
jenkins-daily-20201102000048   Completed   0        0          2020-11-02 02:00:48 +0200 EET   4d        default            <none>
jenkins-daily-20201101000048   Completed   0        0          2020-11-01 02:00:48 +0200 EET   3d        default            <none>
jenkins-daily-20201031000048   Completed   0        0          2020-10-31 02:00:48 +0200 EET   2d        default            <none>
jenkins-daily-20201030000048   Completed   0        0          2020-10-30 02:00:48 +0200 EET   1d        default            <none>
jenkins-daily-20201029000047   Completed   0        0          2020-10-29 02:00:47 +0200 EET   16h       default            <none>
jenkins-test-selector          Completed   0        0          2020-11-25 18:02:29 +0200 EET   28d       default            app.kubernetes.io/name=jenkins

velero restore create jenkins-test-selector-restore --from-backup jenkins-test-selector --namespace-mappings core:jenkins-restore
Restore request "jenkins-test-selector-restore" submitted successfully.
Run `velero restore describe jenkins-test-selector-restore` or `velero restore logs jenkins-test-selector-restore` for more details.

in 40 minutes

date && velero restore get
Fri 27 Nov 2020 10:01:21 AM EET
NAME                            BACKUP                  STATUS       STARTED                         COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
jenkins-test-selector-restore   jenkins-test-selector   InProgress   2020-11-27 09:21:54 +0200 EET   <nil>       0        0          2020-11-27 09:21:54 +0200 EET   <none>

k get all -n jenkins-restore 
NAME                           READY   STATUS    RESTARTS   AGE
pod/jenkins-554866c75b-hjw25   1/1     Running   0          42m

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/jenkins         ClusterIP   10.233.2.178   <none>        80/TCP      42m
service/jenkins-agent   ClusterIP   10.233.53.44   <none>        50000/TCP   42m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jenkins   1/1     1            1           42m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/jenkins-554866c75b   1         1         1       42m
replicaset.apps/jenkins-5747656d46   0         0         0       42m
replicaset.apps/jenkins-57845bdf95   0         0         0       42m
replicaset.apps/jenkins-57cd7d4d8    0         0         0       42m
replicaset.apps/jenkins-5c6fdd87f8   0         0         0       42m
replicaset.apps/jenkins-66ccb89dc7   0         0         0       42m
replicaset.apps/jenkins-6dd85ff744   0         0         0       42m
replicaset.apps/jenkins-7948dc55d7   0         0         0       42m
replicaset.apps/jenkins-7d787bcf9    0         0         0       42m
replicaset.apps/jenkins-868647575d   0         0         0       42m
replicaset.apps/jenkins-8798b6889    0         0         0       42m

k get pvc -n jenkins-restore 
NAME      STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins   Bound    local-pv-90790d64   19Gi       RWO            local-disks    43m

jenkins_heme restored size

k -n jenkins-restore exec pod/jenkins-554866c75b-hjw25 -- du -sh /var/jenkins_home
325M    /var/jenkins_home

jenkins_home original size

k -n core exec jenkins-57845bdf95-bd2k6 -c jenkins -- du -sh /var/jenkins_home
6.9G    /var/jenkins_home

Where is jenkins data from backup?

kubectl logs deployment/velero -n velero
https://gist.github.com/svua/b611f107523c99e3d73d5fcb2223f70e

0 replies

zubron · 2020-11-27T15:00:45Z

zubron
Nov 27, 2020

Thanks for providing the additional information and logs. Looking through them, I can see that the restic restore of the volumes timed out after 4 hours: https://gist.github.com/svua/b611f107523c99e3d73d5fcb2223f70e#file-gistfile1-txt-L1733-L1737 and failed again on the second attempt: https://gist.github.com/svua/b611f107523c99e3d73d5fcb2223f70e#file-gistfile1-txt-L3060-L3065

Given that this is an issue with the restic restore, you will need to look at the logs for the restic daemonset (using kubectl logs -n velero ds/restic) and also inspect the PodVolumeRestore objects that velero creates. Can you please follow the steps here to inspect the PodVolumeRestore objects and let us know what you find: https://velero.io/docs/v1.5/restic/#troubleshooting. Thanks!

0 replies

svua · 2020-11-28T10:47:02Z

svua
Nov 28, 2020
Author

kubectl logs -n velero ds/restic
Found 8 pods, using pod/restic-2zt8k
time="2020-11-25T15:57:46Z" level=info msg="Setting log-level to INFO"
time="2020-11-25T15:57:46Z" level=info msg="Starting Velero restic server v1.5.2 (456eb19668f8da603756353d9179b59b5a7bfa04)" logSource="pkg/cmd/cli/restic/server.go:83"
2020-11-25T15:57:46.977Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
time="2020-11-25T15:57:47Z" level=info msg="Starting controllers" logSource="pkg/cmd/cli/restic/server.go:208"
time="2020-11-25T15:57:47Z" level=info msg="Starting metric server for restic at address [:8085]" logSource="pkg/cmd/cli/restic/server.go:199"
time="2020-11-25T15:57:47Z" level=info msg="Controllers starting..." logSource="pkg/cmd/cli/restic/server.go:250"
2020-11-25T15:57:47.006Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
time="2020-11-25T15:57:47Z" level=info msg="Starting controller" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:76"
time="2020-11-25T15:57:47Z" level=info msg="Starting controller" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:76"
time="2020-11-25T15:57:47Z" level=info msg="Waiting for caches to sync" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:81"
time="2020-11-25T15:57:47Z" level=info msg="Waiting for caches to sync" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:81"
time="2020-11-25T15:57:47Z" level=info msg="Caches are synced" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:85"
time="2020-11-25T15:57:47Z" level=info msg="Caches are synced" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:85"

kubectl get pods -n velero
NAME                      READY   STATUS    RESTARTS   AGE
restic-2zt8k              1/1     Running   0          2d18h
restic-8wk48              1/1     Running   0          2d18h
restic-btrsx              1/1     Running   0          2d18h
restic-l76fn              1/1     Running   1          2d18h
restic-nwj6g              1/1     Running   1          2d18h
restic-qjjs7              1/1     Running   0          2d18h
restic-v9npn              1/1     Running   0          2d18h
restic-wjfg9              1/1     Running   0          2d18h
velero-84b455d4db-qbtpb   1/1     Running   0          2d18h

velero restic repo get
NAME                 STATUS   LAST MAINTENANCE
core-default-2jj2s   Ready    2020-11-25 18:02:38 +0200 EET

velero restic repo get core-default-2jj2s -o yaml
apiVersion: velero.io/v1
kind: ResticRepository
metadata:
  creationTimestamp: "2020-11-25T16:02:38Z"
  generateName: core-default-
  generation: 3
  labels:
    velero.io/storage-location: default
    velero.io/volume-namespace: core
  managedFields:
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
        f:labels:
          .: {}
          f:velero.io/storage-location: {}
          f:velero.io/volume-namespace: {}
      f:spec:
        .: {}
        f:backupStorageLocation: {}
        f:maintenanceFrequency: {}
        f:resticIdentifier: {}
        f:volumeNamespace: {}
      f:status:
        .: {}
        f:lastMaintenanceTime: {}
        f:phase: {}
    manager: velero-server
    operation: Update
    time: "2020-11-25T16:02:38Z"
  name: core-default-2jj2s
  namespace: velero
  resourceVersion: "154244802"
  selfLink: /apis/velero.io/v1/namespaces/velero/resticrepositories/core-default-2jj2s
  uid: 0fa7ebda-a85b-4052-86db-784fe7b329ac
spec:
  backupStorageLocation: default
  maintenanceFrequency: 168h0m0s
  resticIdentifier: s3:http://minio-service.cold-data.svc.cluster.local:9000/velero/restic/core
  volumeNamespace: core
status:
  lastMaintenanceTime: "2020-11-25T16:02:38Z"
  phase: Ready

velero backup describe jenkins-test-selector
Name:         jenkins-test-selector
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.3
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  app.kubernetes.io/name=jenkins

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2020-11-25 18:02:29 +0200 EET
Completed:  2020-11-25 18:07:45 +0200 EET

Expiration:  2020-12-25 18:02:29 +0200 EET

Total items to be backed up:  26
Items backed up:              26

Velero-Native Snapshots: <none included>

Restic Backups (specify --details for more information):
  Completed:  7

velero backup logs jenkins-test-selector
https://gist.github.com/svua/23e44512111f8dbfbc8c1a59184edd9a

velero restore get
NAME                            BACKUP                  STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
jenkins-test-selector-restore   jenkins-test-selector   PartiallyFailed   2020-11-27 09:21:54 +0200 EET   2020-11-27 13:21:54 +0200 EET   1        0          2020-11-27 09:21:54 +0200 EET   <none>

velero restore describe jenkins-test-selector-restore
Name:         jenkins-test-selector-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs jenkins-test-selector-restore' for more information)

Started:    2020-11-27 09:21:54 +0200 EET
Completed:  2020-11-27 13:21:54 +0200 EET

Errors:
  Velero:   timed out waiting for all PodVolumeRestores to complete
  Cluster:    <none>
  Namespaces: <none>

Backup:  jenkins-test-selector

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  core=jenkins-restore

Label selector:  <none>

Restore PVs:  auto

Restic Restores (specify --details for more information):
  New:  6

velero restore logs jenkins-test-selector-restore
https://gist.github.com/svua/0efbb9a70a739825cec2093ff92ddae4

kubectl -n velero get podvolumebackups
NAME                          AGE
jenkins-test-selector-7rxc5   2d18h
jenkins-test-selector-lpwwn   2d18h
jenkins-test-selector-q656t   2d18h
jenkins-test-selector-qnjcc   2d18h
jenkins-test-selector-v2g7b   2d18h
jenkins-test-selector-wvh6j   2d18h
jenkins-test-selector-zx25m   2d18h

kubectl -n velero get podvolumebackups -l velero.io/backup-name=jenkins-test-selector -o yaml
https://gist.github.com/svua/ef092c96bc2f82b162161b381d72d3b2

kubectl -n velero get podvolumerestores
NAME                                  AGE
jenkins-test-selector-restore-2h7mp   26h
jenkins-test-selector-restore-2zj2d   26h
jenkins-test-selector-restore-9jnwh   26h
jenkins-test-selector-restore-hv7q9   26h
jenkins-test-selector-restore-r5tcb   26h
jenkins-test-selector-restore-rsmkq   26h

kubectl -n velero get podvolumerestores -l velero.io/restore-name=jenkins-test-selector-restore -o yaml
https://gist.github.com/svua/4cb11c08a320d97c0beefe04d689ced8

0 replies

zubron · 2020-11-30T12:44:14Z

zubron
Nov 30, 2020

Thanks again for providing the additional logs. Unfortunately, the command I gave you to get the restic logs only provided the logs for one of the restic pods, not all the pods in the DaemonSet. Can you check the logs for all the restic DaemonSet pods? I can't see anything that stands out in the other logs, and it will be the restic pods that process the PodVolumeRestores which aren't making progress.

0 replies

svua · 2020-11-30T17:29:12Z

svua
Nov 30, 2020
Author

All pods have the same logs, except for one

kubectl -n velero logs restic-v9npn
time="2020-11-25T15:57:47Z" level=info msg="Setting log-level to INFO"
time="2020-11-25T15:57:47Z" level=info msg="Starting Velero restic server v1.5.2 (456eb19668f8da603756353d9179b59b5a7bfa04)" logSource="pkg/cmd/cli/restic/server.go:83"
2020-11-25T15:57:47.167Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
time="2020-11-25T15:57:47Z" level=info msg="Starting controllers" logSource="pkg/cmd/cli/restic/server.go:208"
time="2020-11-25T15:57:47Z" level=info msg="Starting metric server for restic at address [:8085]" logSource="pkg/cmd/cli/restic/server.go:199"
time="2020-11-25T15:57:47Z" level=info msg="Controllers starting..." logSource="pkg/cmd/cli/restic/server.go:250"
2020-11-25T15:57:47.202Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
time="2020-11-25T15:57:47Z" level=info msg="Starting controller" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:76"
time="2020-11-25T15:57:47Z" level=info msg="Waiting for caches to sync" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:81"
time="2020-11-25T15:57:47Z" level=info msg="Starting controller" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:76"
time="2020-11-25T15:57:47Z" level=info msg="Waiting for caches to sync" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:81"
time="2020-11-25T15:57:47Z" level=info msg="Caches are synced" controller=pod-volume-backup logSource="pkg/controller/generic_controller.go:85"
time="2020-11-25T15:57:47Z" level=info msg="Caches are synced" controller=pod-volume-restore logSource="pkg/controller/generic_controller.go:85"
time="2020-11-25T16:02:38Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-7rxc5 namespace=velero
time="2020-11-25T16:02:47Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-7rxc5 namespace=velero
time="2020-11-25T16:02:47Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-wvh6j namespace=velero
time="2020-11-25T16:02:54Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-wvh6j namespace=velero
time="2020-11-25T16:02:54Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-v2g7b namespace=velero
time="2020-11-25T16:03:01Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-v2g7b namespace=velero
time="2020-11-25T16:03:01Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-zx25m namespace=velero
time="2020-11-25T16:03:09Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-zx25m namespace=velero
time="2020-11-25T16:03:09Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-q656t namespace=velero
time="2020-11-25T16:03:09Z" level=info msg="Looking for most recent completed pod volume backup for this PVC" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:342" name=jenkins-test-selector-q656t namespace=velero pvcUID=681fc57c-84ac-4c4c-8b0f-1cb5a92f0329
time="2020-11-25T16:03:09Z" level=info msg="No completed pod volume backup found for PVC" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:372" name=jenkins-test-selector-q656t namespace=velero pvcUID=681fc57c-84ac-4c4c-8b0f-1cb5a92f0329
time="2020-11-25T16:03:09Z" level=info msg="No parent snapshot found for PVC, not using --parent flag for this backup" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:283" name=jenkins-test-selector-q656t namespace=velero
time="2020-11-25T16:07:32Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-q656t namespace=velero
time="2020-11-25T16:07:32Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-qnjcc namespace=velero
time="2020-11-25T16:07:38Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-qnjcc namespace=velero
time="2020-11-25T16:07:38Z" level=info msg="Backup starting" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:191" name=jenkins-test-selector-lpwwn namespace=velero
time="2020-11-25T16:07:44Z" level=info msg="Backup completed" backup=velero/jenkins-test-selector controller=pod-volume-backup logSource="pkg/controller/pod_volume_backup_controller.go:332" name=jenkins-test-selector-lpwwn namespace=velero

0 replies

ashish-amarnath · 2020-11-30T20:15:40Z

ashish-amarnath
Nov 30, 2020

@svua Can you please do a kubectl describe pod on the pod that is failing to be restored?
It may be the case that the restic-restore-helper image might be missing.

0 replies

svua · 2020-12-01T05:30:59Z

svua
Dec 1, 2020
Author

All pods have the same description

Pod that is failing

Name:         restic-v9npn
Namespace:    velero
Priority:     0
Node:         storage1/172.16.1.10
Start Time:   Wed, 25 Nov 2020 17:57:46 +0200
Labels:       component=velero
              controller-revision-hash=b5fbf9bc4
              name=restic
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           10.233.82.73
IPs:
  IP:           10.233.82.73
Controlled By:  DaemonSet/restic
Containers:
  restic:
    Container ID:  docker://6c8ee78a24726a23b8d986dc8f0fa3722db30d4edbfd0420f2281bdcba01ffce
    Image:         velero/velero:v1.5.2
    Image ID:      docker-pullable://velero/velero@sha256:e248548c6787f5451e1ba42a48ad4890ce1fc10aa6f5efca9aeb3aa6cda396ff
    Port:          <none>
    Host Port:     <none>
    Command:
      /velero
    Args:
      restic
      server
      --features=
    State:          Running
      Started:      Wed, 25 Nov 2020 17:57:47 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment:
      NODE_NAME:                        (v1:spec.nodeName)
      VELERO_NAMESPACE:                velero (v1:metadata.namespace)
      VELERO_SCRATCH_DIR:              /scratch
      GOOGLE_APPLICATION_CREDENTIALS:  /credentials/cloud
      AWS_SHARED_CREDENTIALS_FILE:     /credentials/cloud
      AZURE_CREDENTIALS_FILE:          /credentials/cloud
      ALIBABA_CLOUD_CREDENTIALS_FILE:  /credentials/cloud
    Mounts:
      /credentials from cloud-credentials (rw)
      /host_pods from host-pods (rw)
      /scratch from scratch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from velero-token-9pkh5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  host-pods:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  
  scratch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  cloud-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloud-credentials
    Optional:    false
  velero-token-9pkh5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  velero-token-9pkh5
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:          <none>

Another pod

Name:         restic-2zt8k
Namespace:    velero
Priority:     0
Node:         ch4/172.16.1.14
Start Time:   Wed, 25 Nov 2020 17:57:46 +0200
Labels:       component=velero
              controller-revision-hash=b5fbf9bc4
              name=restic
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           10.233.91.36
IPs:
  IP:           10.233.91.36
Controlled By:  DaemonSet/restic
Containers:
  restic:
    Container ID:  docker://cdb89991c1fb6642356b0fafcab5099b636f15bd97dee1b8db000611e525b1e5
    Image:         velero/velero:v1.5.2
    Image ID:      docker-pullable://velero/velero@sha256:e248548c6787f5451e1ba42a48ad4890ce1fc10aa6f5efca9aeb3aa6cda396ff
    Port:          <none>
    Host Port:     <none>
    Command:
      /velero
    Args:
      restic
      server
      --features=
    State:          Running
      Started:      Wed, 25 Nov 2020 17:57:46 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment:
      NODE_NAME:                        (v1:spec.nodeName)
      VELERO_NAMESPACE:                velero (v1:metadata.namespace)
      VELERO_SCRATCH_DIR:              /scratch
      GOOGLE_APPLICATION_CREDENTIALS:  /credentials/cloud
      AWS_SHARED_CREDENTIALS_FILE:     /credentials/cloud
      AZURE_CREDENTIALS_FILE:          /credentials/cloud
      ALIBABA_CLOUD_CREDENTIALS_FILE:  /credentials/cloud
    Mounts:
      /credentials from cloud-credentials (rw)
      /host_pods from host-pods (rw)
      /scratch from scratch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from velero-token-9pkh5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  host-pods:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  
  scratch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  cloud-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloud-credentials
    Optional:    false
  velero-token-9pkh5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  velero-token-9pkh5
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:          <none>

0 replies

DonMarik · 2020-12-16T12:52:03Z

DonMarik
Dec 16, 2020

@svua how have you even got this jenkins home volume backup working? Do you stop a Jenkins instance before running the backups?

I have tried to perform Jenkins backup with velero+restic multiple times and it failed every single time - jenkins is running all the time and I tested backups on a live system, but since the Jenkins app was running all the time and had not been stopped before running a backup, restic could not back up the jenkins home volume as it was like 170 GB of data and constantly changing due to running Jenkins Jobs.

Since we are not willing to stop the Jenkins instance at all, I think that the best option would be to copy over to a separate volume only the data that we need to restore Jenkins properly and then run restic for this new volume.
Any idea how to do it properly with velero and the restic plugin?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore fails due to Restic timeout #3250

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Restore fails due to Restic timeout #3250

svua Nov 25, 2020

Replies: 9 comments

zubron Nov 26, 2020

svua Nov 27, 2020 Author

zubron Nov 27, 2020

svua Nov 28, 2020 Author

zubron Nov 30, 2020

svua Nov 30, 2020 Author

ashish-amarnath Nov 30, 2020

svua Dec 1, 2020 Author

DonMarik Dec 16, 2020

svua
Nov 25, 2020

zubron
Nov 26, 2020

svua
Nov 27, 2020
Author

zubron
Nov 27, 2020

svua
Nov 28, 2020
Author

zubron
Nov 30, 2020

svua
Nov 30, 2020
Author

ashish-amarnath
Nov 30, 2020

svua
Dec 1, 2020
Author

DonMarik
Dec 16, 2020