Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] cluster snapshot backup always InProgress:cannot list resource "volumesnapshotclasses" #3946

Closed
JashBook opened this issue Jun 26, 2023 · 0 comments · Fixed by #3947
Closed
Assignees
Labels
bug kind/bug Something isn't working
Milestone

Comments

@JashBook
Copy link
Collaborator

JashBook commented Jun 26, 2023

Describe the bug
cluster snapshot backup always InProgress:job Completed,but backup is Waiting for the job hook-pre-1628cdd8-backup-default-pg-cluster-20230626103659 creation to complete.
Failed to watch v1.VolumeSnapshotClass: failed to list v1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:default:kubeblocks" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope

kbcli version
Kubernetes: v1.25.9-eks-0a21954
KubeBlocks: 0.6.0-alpha.22
kbcli: v0.6.0-alpha.22

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
  2. snapshot backup
  3. See error
kubectl get pod,backup,volumesnapshot -l app.kubernetes.io/instance=pg-cluster
NAME                          READY   STATUS    RESTARTS   AGE
pod/pg-cluster-postgresql-0   5/5     Running   0          36m
pod/pg-cluster-postgresql-1   5/5     Running   0          36m

NAME                                                                           TYPE       STATUS       TOTAL-SIZE   DURATION   CREATE-TIME            COMPLETION-TIME
backup.dataprotection.kubeblocks.io/b385d182-pg-cluster-default-logfile        logfile    Failed                               2023-06-26T02:38:02Z   
backup.dataprotection.kubeblocks.io/backup-default-pg-cluster-20230626103659   snapshot   InProgress                           2023-06-26T02:37:00Z   
kubectl get job,pod
NAME                                                                       COMPLETIONS   DURATION   AGE
job.batch/hook-pre-1628cdd8-backup-default-pg-cluster-20230626103659       1/1           6s         17m

NAME                                                                  READY   STATUS      RESTARTS         AGE
pod/hook-pre-1628cdd8-backup-default-pg-cluster-20230626103659nz7jz   0/1     Completed   0                17m

kubectl describe backup.dataprotection.kubeblocks.io/backup-default-pg-cluster-20230626103659 
Name:         backup-default-pg-cluster-20230626103659
Namespace:    default
Labels:       app.kubernetes.io/instance=pg-cluster
              app.kubernetes.io/managed-by=kubeblocks
              apps.kubeblocks.io/component-name=postgresql
              dataprotection.kubeblocks.io/backup-type=snapshot
              kubeblocks.io/backup-protection=retain
Annotations:  dataprotection.kubeblocks.io/target-pod-name: pg-cluster-postgresql-0
              kubeblocks.io/cluster-snapshot:
                {"metadata":{"name":"pg-cluster","namespace":"default","creationTimestamp":null},"spec":{"clusterDefinitionRef":"postgresql","clusterVersi...
API Version:  dataprotection.kubeblocks.io/v1alpha1
Kind:         Backup
Metadata:
  Creation Timestamp:  2023-06-26T02:37:00Z
  Finalizers:
    dataprotection.kubeblocks.io/finalizer
  Generation:        1
  Resource Version:  107921326
  UID:               1628cdd8-4867-4114-af7e-3360d8bd650f
Spec:
  Backup Policy Name:  pg-cluster-postgresql-backup-policy
  Backup Type:         snapshot
Status:
  Expiration:       2023-07-03T02:37:00Z
  Phase:            InProgress
  Start Timestamp:  2023-06-26T02:37:00Z
Events:
  Type    Reason                                                                  Age   From               Message
  ----    ------                                                                  ----  ----               -------
  Normal  CreatingJob-hook-pre-1628cdd8-backup-default-pg-cluster-20230626103659  18m   backup-controller  Waiting for the job hook-pre-1628cdd8-backup-default-pg-cluster-20230626103659 creation to complete.
kubectl logs jobs/hook-pre-1628cdd8-backup-default-pg-cluster-20230626103659 
CHECKPOINT

log kubeblocks

2023-06-26T02:52:06.227Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "b385d182-pg-cluster-default-logfile-28129132-6jwnt", "up to level": 2}
2023-06-26T02:52:06.233Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "b385d182-pg-cluster-default-logfile-28129132-6jwnt", "up to level": 2}
W0626 02:52:13.549926       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:default:kubeblocks" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope
E0626 02:52:13.549983       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.VolumeSnapshotClass: failed to list *v1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:default:kubeblocks" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope
W0626 02:52:49.503855       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:default:kubeblocks" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope
E0626 02:52:49.504101       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.VolumeSnapshotClass: failed to list *v1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:default:kubeblocks" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope
2023-06-26T02:53:00.146Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "b385d182-pg-cluster-default-logfile-28129133-m84d7", "up to level": 2}
2023-06-26T02:53:00.149Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "5de0dc8d-cluster-sbrkvj-defaul-logfile-28129133-4x4tq", "up to level": 2}
2023-06-26T02:53:00.162Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "b385d182-pg-cluster-default-logfile-28129133-m84d7", "up to level": 2}
2023-06-26T02:53:00.162Z	INFO	eventhandler.EnqueueRequestForAncestor	cloud not find ancestor object	{"source object gvk": "/, Kind=", "name": "b385d182-pg-cluster-default-logfile-28129133-m84d7", "up to level": 2}

Expected behavior
cluster snapshot backup success.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@JashBook JashBook added the kind/bug Something isn't working label Jun 26, 2023
@JashBook JashBook added this to the Release 0.6.0 milestone Jun 26, 2023
@wangyelei wangyelei linked a pull request Jun 26, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants