Debugging warnings in pss workflow

[biswajit-9776]

Pull Request Template for Kubeflow manifests Issues

✏️ A brief description of the changes

I pushed a PR just to view workflows here in CICD

📦 List any dependencies that are required for this change

My PR depends on #

🐛 If this PR is related to an issue, please put the link to the issue here.

The following issues are related, because ...

✅ Contributor checklist

• Make sure you have tested with kustomize. See Installation Prerequisites
• All the commits have been signed-off (To pass the DCO check)

---

You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

[juliusvonkohout]

Loogs good, now you just need to create patches to fix

Run ./tests/gh-actions/enable_restricted_PSS.sh
...
Patching the PSS-restricted labels for namespace oauth2-proxy...
Warning: existing pods in namespace "oauth2-proxy" violate the new PodSecurity enforce level "restricted:latest"
Warning: oauth2-proxy-5bcc78bdc9-7f24j: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/oauth2-proxy patched
Patching the PSS-restricted labels for namespace kubeflow...
Warning: existing pods in namespace "kubeflow" violate the new PodSecurity enforce level "restricted:latest"
namespace/kubeflow patched
Warning: cache-server-5dbfbbdc95-bk6rl (and 12 other pods): unrestricted capabilities, runAsNonRoot != true, runAsUser=0, seccompProfile
Warning: cache-server-64cbf5d9f6-6h8nh (and 9 other pods): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, runAsUser=0, seccompProfile
Warning: kubeflow-pipelines-profile-controller-7b7b8f44f7-ktpxd (and 1 other pod): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
Warning: metacontroller-0: seccompProfile

[biswajit-9776]

- name: Apply patches to clear warnings
      run: |
        DIRECTORY="contrib/security/PSS/patches"
        for file in "$DIRECTORY"/*.yaml; do
          echo "Patching file: $file"
          KIND=$(kubectl get -f "$file" -o jsonpath='{.kind}')
          NAME=$(kubectl get -f "$file" -o jsonpath='{.metadata.name}')
          NAMESPACE=$(kubectl get -f "$file" -o jsonpath='{.metadata.namespace}')
          # Apply the patch
          kubectl get "$KIND" "$NAME" -n "$NAMESPACE" &> /dev/null
          if [ $? -eq 0 ]; then
            kubectl patch "$KIND" "$NAME" -n "$NAMESPACE" --patch-file "$file"
          fi
        done

This job in our workflow was added to fix the warnings. Locally it does so but I don't see why it seems to not work here.
Let me restart some of the deployments to see if that works.

[juliusvonkohout]

I also see for example

manifests/contrib/security/PSS/patches/oauth2-proxy.yaml

Line 7 in da0255f

replicas: 2

which should not be the case. It seems to patch the replicas.

[juliusvonkohout]

maybe you need to wait for all deployments to restart (pods) after the patches and become ready. But they should restart automatically.

[biswajit-9776]

I also see for example

manifests/contrib/security/PSS/patches/oauth2-proxy.yaml

Line 7 in da0255f

replicas: 2

which should not be the case. It seems to patch the replicas.

Ohh, this line isn't needed so I'll remove it

[biswajit-9776]

@juliusvonkohout the kubeflow pods fail to restart successfully, even after 10 minutes. Any suggestions?

[juliusvonkohout]

maybe some of them are failing due to the security context. maybe they are in a crashloopbackoff

[juliusvonkohout]

Maybe you should add the readiness wait to your patch loop and test one by one this way

[biswajit-9776]

@juliusvonkohout Hey, how much wait time should I set for each pod? Apparently 180s isn't enough.

[biswajit-9776]

Even though the istio-cni uses init containers with proper securityContext attributes, it misses the seccompProfile attribute, that may lead to the following warnings as of now:

Run ./tests/gh-actions/enable_restricted_PSS.sh
Patching the PSS-restricted labels for namespace istio-system...
namespace/istio-system patched
Patching the PSS-restricted labels for namespace auth...
namespace/auth patched
Patching the PSS-restricted labels for namespace cert-manager...
namespace/cert-manager patched
Patching the PSS-restricted labels for namespace oauth2-proxy...
namespace/oauth2-proxy patched
Patching the PSS-restricted labels for namespace kubeflow...
Warning: existing pods in namespace "kubeflow" violate the new PodSecurity enforce level "restricted:latest"
Warning: cache-server-6c656d7988-v[4](https://github.com/kubeflow/manifests/actions/runs/10868317592/job/30158117073?pr=2866#step:10:5)nzx (and 12 other pods): seccompProfile
Warning: metadata-grpc-deployment-8496ffb98b-wxc6p (and 1 other pod): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/kubeflow patched

The warnings for baseline label are gone as of now.

[juliusvonkohout]

In the Istio initcontainer configmap or so in the Istio-system namespace you might be able to set the seccomp profile.

[juliusvonkohout]

You can also try to set the securitycontext on the pod instead of container level.

[biswajit-9776]

The patches targeting deployments with containers, would that be unnecessary? As just the initContainer is failing the seccompProfile...

[biswajit-9776]

Hey @juliusvonkohout, are there any patches to templates of configmap in the source code that I could use as a reference? A simple addition of seccompProfile to the sidecar template of the configmap replaces all other data with my patched attribute that I wouldn't want.

[juliusvonkohout]

The patches targeting deployments with containers, would that be unnecessary? As just the initContainer is failing the seccompProfile...

You have to test that.

[juliusvonkohout]

Hey @juliusvonkohout, are there any patches to templates of configmap in the source code that I could use as a reference? A simple addition of seccompProfile to the sidecar template of the configmap replaces all other data with my patched attribute that I wouldn't want.

I would have to check that. In the worst case you have to patch it with sed or so.

[biswajit-9776]

Okay, so I have used an ugly sed command for the time being @juliusvonkohout. It does solve the istio-containers failing the labels.
There's only the metadata-grpc-deployment and I believe the ml-pipeline that are currently failing the warnings even after the patches which I also see locally, as they go to CrashLoopBackOff and their previously healthy non-terminated pods generate the warnings.

[juliusvonkohout]

Please remove all the istio initcontainer patches from all
deployment patches. They are injected at runtime and changed by the general istio configuration you modified with sed

[biswajit-9776]

initContainers have been removed in the commit e40311f

[juliusvonkohout]

Please remove the debugging parts, such that we can merge soon. Let's just finish these three comments and merge. This PR is getting too big and we should continue in new ones.

[biswajit-9776]

Debugging parts along with patch for dynamic namespaces have also been removed in the commit e40311f. I believe that can be done in a new PR to keep things simple.

[juliusvonkohout]

This looks good :-) according to the workflow results it seems to work, but please verify it manually.

[biswajit-9776]

Yes, I did run it manually and opened up the configmap to observe the changes in place as we want. Also, I have removed the comments which I can later add as commits for dynamic namespace.

[biswajit-9776]

The ml-pipeline pod in my local system does seem to fail with CrashLoopBackOff as I open it up. Maybe I need to increase my fs.inotify.max_user_{instances, watches} .

[lampajr]

@biswajit-9776 tests look much better now, the only warning I see at the moment is:

Warning: ml-pipeline-bf9f88745-qzmm9: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile

[biswajit-9776]

@biswajit-9776 tests look much better now, the only warning I see at the moment is:

Warning: ml-pipeline-bf9f88745-qzmm9: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile

Yes @lampajr, it also looks the same in my local system. Upon opening the pod logs, it doesn't fail the PSS warnings though. It's failing to start up again.

[juliusvonkohout]

/lgtm
/approve

lets continue in follow up PRs.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [juliusvonkohout]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment