Backup requires cluster scope to succeed, even when installed without clusterAdministrator #5156
Comments
|
@reasonerjt - I'm wondering if you have any thoughts/ideas on this issue. I've done some more digging, and I haven't been able to make Velero work without granting It appears that Velero requires the service-account to have permissions to list ALL resources at cluster-scope, even if it's configured to only backup resources in a single namespace. Is there some way around this? I first tried to see if I could control the backup and prevent it from accessing resources that might be at cluster scope, or resources which it might not have permissions to list/get. For both of these cases below, I continued to get the permission error listed in the original description (eg: error getting ClusterRoleBindings at cluster-scope)
Then, on a dev environment (where I have permissions to create ClusterRoleBindings) I tried to see if I could proceed with the backup by creating a ClusterRoleBinding that only allowed for
Any thoughts? |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Closing the stale issue. |
|
@reasonerjt Has there been any development on this? I'm noticing a similar issue trying to use velero v1.10.0 in Azure |
|
Hi - Can this JIRA be re-opened? |
|
I think this is related to k8s ServiceAccount resource. There is a Velero plugin applies to ServiceAccount. The plugin will go through all ClusterRoleBinding and ClusterRole, then return the related ClusterRoleBinding and ClusterRole to the ServiceAccount as additional backup items. If it's possible, please also exclude the ServiceAccount resource from the backup. |
|
That is a bit hacky solution, and there may also be other resources that need cluster-scope resources access permission. @fryz Could you give some details about your environment? Velero is designed to work with the cluster administrator's permission. Is it possible to let Velero have the administrator's permission, and only back resources in specific namespace? |
|
@fryz I also saw that ClusterRole and ClusterRoleBinding are also included in the Role permissions, so you are just trying to back namespaced-scope resource up, right? |
|
Hi @blackpiglet, I work closely with @fryz and can try to provide some context on our environment. We are deploying an application in an enterprise Kubernetes environment in which we only have access to a single namespace as the application vendor. The enterprise controls do not allow us to use any ClusterRoles or ClusterRoleBindings, especially ClusterAdministrator. We are hoping to be able to use Velero in a namespace-scoped fashion so we can backup and restore our application. Thanks for the tip about excluding the service accounts. I will try to get a test setup in the next few days to try your suggestion. @fryz is currently out but should be back next week as well with more information. Thanks for the help! |
|
I did some tests to install Velero without any cluster-scoped resource permissions, but the Velero server failed to start. The failure is: I think this is due to the Velero server needing to confirm the namespace it runs in exists before spinning up the controllers. velero/pkg/cmd/server/server.go Lines 412 to 417 in b7cc62d It seems that error didn't happen in your cluster. Could you help to confirm your environment's Velero permission setting? It looks like it's inevitable to be involved with some cluster-scoped resource access. |
|
Hey @blackpiglet - thanks for the work and attempts to help us get this working. Quick note - when we originally ran into this issue, we were using Velero 1.10 and the 2.32 version of the helm chart. My plan was to take the first step and confirm if this works on the latest version using your workaround re: ServiceAccounts above (sounds like it doesn't work based on your latest comment?). Then, if it doesn't work, I was going to try on the versions that we are using to see if the workaround works there. |
|
Hey @blackpiglet Just quickly wanted to let you know that when I tried installing/configuring using the setup above only using Role/RoleBindings (on the 1.10 version of velero with the 2.32 version of the helm-chart), I ran into issues with Velero during server initialization because it's looking for the Velero Custom Resources at cluster scope: So I think from our end, I wasn't able to get to the point where I could take a backup without granting the velero-server ServiceAccount cluster-scoped permissions on the Velero Cluster Resources. |
|
@fryz I think this piece of code is used to mark the InProgress backups as failed during the Velero server start. |
|
@fryz |
What steps did you take and what happened:
Due to restrictions on the k8s environment I am running in, I cannot use ClusterRoleBindings because it grants cluster scope. To address this, I have installed Velero (using the helm chart, values.yaml file provided below) with the following configuration:
rbac.clusterAdministrator, and therefore, the ClusterRoleBinding is not generated and not appliedapp), and we expect to only backup resources within that namespace (eg: not CRDs, PVs, etc.)With the aforementioned configuration, I try to take a backup:
And I get the following in the backup-controller logs:
With the specific failure being:
Even when I attempt to exclude CRBs as a resource from the backup, I get the same error:
What did you expect to happen:
Backup's should not require cluster-scope to backup resources when disabled, or I should be able to exclude resources that require cluster-scope from the backup to get the backup to succeed.
The following information will help us better understand what's going on:
I'm not going to attach the debug output because it contains the logs and state of all pods/resources in the cluster. Let me know if you need specific information and I can provide.
Anything else you would like to add:
Helm Values File:
Environment:
velero version): v1.9.0velero client config get features): NOT SETkubectl version):Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:21:56Z", GoVersion:"go1.18.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.12-eks-a64ea69", GitCommit:"d4336843ba36120e9ed1491fddff5f2fec33eb77", GitTreeState:"clean", BuildDate:"2022-05-12T18:29:27Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
/etc/os-release):Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
The text was updated successfully, but these errors were encountered: