RBAC: access denied - cannot access notebook created in my namespace

[bgalvao]

Checks

• I have searched the existing issues.
• This issue is NOT specific to the CLI. (If so, please open an issue on the CLI repo)

deployKF Version

0.1.4

Kubernetes Version

Client Version: v1.28.8
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1-eks-b9c9ed7

Description

I was trying to access a notebook I created, but surprisingly I was denied access to the notebook in my own namespace / profile.

Relevant Logs

Don't know where to look for relevant logs (some kyverno pod?), but I'm happy to be instructed and update this field with the relevant logs.

deployKF Values (Optional)

deploykf_auth:
  dex:
    connectors:
      - type: google
        id: google
        name: Google
        config: (...)

deploykf_profiles_generator:
  users:
    - id: bernardo-galvao
      email: "bernardo.galvao@pumpkinai.com"
  profiles:
    - name: bernardo-galvao
      members:
        - user: bernardo-galvao
          access:
            role: edit
            notebookAccess: true

[bgalvao]

Additional context:

• using EKS
• managing access via Access Entries of EKS
• my user is not in the aws-auth ConfigMap

I think fixing that should fix the issue

[thesuperzapper]

@bgalvao are the notebooks the only places you are seeing "RBAC: Access Denied" errors?

For example, is this user able to browse the dashboard in that namespace, and view pipelines?

[bgalvao]

Hey @thesuperzapper, yes that is the only place.

• I can navigate to volumes, create volumes.
• Can navigate to Tensorboard

(I disabled KF pipelines for this instance of deployKF as my team does not use them.)

---

Update:

• Tried modifying the owner in the spec of the profile to my own email (instead of admin@example.com). Did not work.

  

• Also, my profile is not the owner of any namespaces regardless. Not even the namespace attributed to my profile / email.

• added my user to aws-auth configmap. Also did not fix anything.

---

Is there any pod which logs I should take a look at?
Was I too hasty with using k8s 1.29?

[bgalvao]

@thesuperzapper
Big thanks to @gilcardoai for finding the solution. (Expand for diffs)

ns-owner-access-istio AuthorizationPolicy

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  annotations:
    role: admin
    user: admin@example.com
  name: ns-owner-access-istio
  namespace: bernardo-galvao
spec:
  rules:
    - when:
        - key: request.headers[kubeflow-userid]
          values:
-           - admin@example.com
+           - bernardo.galvao@pumpkinai.com
    - when:
        - key: source.namespace
          values:
            - bernardo-galvao
    - to:
        - operation:
            paths:
              - /healthz
              - /metrics
              - /wait-for-drain
    - from:
        - source:
            principals:
              - cluster.local/ns/kubeflow/sa/notebook-controller-service-account
      to:
        - operation:
            methods:
              - GET
            paths:
              - '*/api/kernels'

Profile

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  finalizers:
    - profile-finalizer
  name: bernardo-galvao
spec:
  owner:
    kind: User
-   name: admin@example.com
+   name: bernardo.galvao@pumpkinai.com

---

That being said, I have not looked into the internals of deployKF, but there seems to be an override Authorization policy that is perhaps supposed to be updated with an authorize action for my user (bernardo.galvao@pumpkinai.com), because all it has is a DENY action for the only email that was present before as the owner of the Profile: admin@example.com:

ns-owner-access-istio--override

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ns-owner-access-istio--override
  namespace: bernardo-galvao
spec:
  action: DENY
  rules:
    - from:
        - source:
            notPrincipals:
              - cluster.local/ns/deploykf-istio-gateway/sa/deploykf-gateway
      when:
        - key: request.headers[kubeflow-userid]
          values:
            - admin@example.com

Also, when I tried to add my own email as the ownerEmail of profile, like this

deploykf_profiles_generator:
  users:
    - id: bernardo-galvao
      email: "bernardo.galvao@pumpkinai.com"
  profiles:
    - name: bernardo-galvao
      ownerEmail: bernardo.galvao@pumpkinai.com
      members:
        - user: bernardo-galvao
          access:
            role: edit
            notebookAccess: true

that is not something that is allowed:

(profile name: 'bernardo-galvao') may not include users with same email as profile owner, but user 'bernardo-galvao' at index 0 has email 'bernardo.galvao@pumpkinai.com'

meaning that it uses the default admin@example.com.

---

Sorry for the long comment, I wanted to be as detailed as possible. The project deployKF is awesome and I want to see it grow!

Update: I finally found the issue kubeflow/dashboard#33 👍🏼

[thesuperzapper]

@bgalvao I know it's a bit confusing you should NOT set yourself as "owner" of any profiles.

You should not even change it from admin@example.com, just remove that account from staticPasswords and assign yourself as a "member" with edit access.

I will probably actually remove the ability to set an owner, because it just causes to many problems:

1. You can't update the owner of a profile
2. Profile owners can log in to stuff like "minio" and "argo server" UIs
3. If someone tries to use the "manage contributors" page of the dashboard, they will break the profiles.

[thesuperzapper]

@bgalvao either way, you should never be messing with the AuthorizationPolicies.

Can you confirm if you were previously trying to set the profile owners with a deploKF value?

Also, you should revert those changes by re-generating your manifests, and syncing (with prune enabled).

[bgalvao]

Can you confirm if you were previously trying to set the profile owners with a deploKF value?

Yeah, but some validator did not allow to use an email that was already set for a user, so I reverted it.

The fix for the behavior I wanted anyway is in the workaround mentioned in kubeflow/dashboard#33, so it is good that way for us. Thanks :)

(imo this can be closed)

[thesuperzapper]

@bgalvao I strongly believe your issue will have been related to not pruning correctly.

I have updated (#123) the sync script to require pruning (previously if you did not respond to the prompt, it would skip pruning, which could break authentication is strange ways).

Update to the latest version of the sync script found in main:

• https://github.com/deployKF/deployKF/blob/main/scripts/sync_argocd_apps.sh