[Launcher] support non-root containers in v2 compatible mode

[Bobgy]

Currently, because the launcher writes input artifacts to paths like:

• /gcs/xxx
• /minio/xxx
• /s3/xxxx

These paths are not accessible by non-Root users by default.

When using a component with non-Root image, launcher fails when preparing input/output artifacts.

Because /gcs/xxx is currently a contract for KFP v2 python component wrappers, we cannot change to a different path like /tmp/gcs/xxx etc.

[Ark-kun]

It's possible to specify the user that the container should run under. Backend can auto-set that on all containers.

[Bobgy]

@Ark-kun do you mean our backend sets all containers to run as root by default?

[Nagarajj]

@Bobgy What is the plan to support non-root containers ? it is difficult to do any PoC with V2 in an enterprise setup due to this. Thanks.

[Bobgy]

Hi @Nagarajj, this isn't on my priority list right now.
So welcome contributions on this! Feel free to discuss if you need any help.
Or if there are more people chiming in that this is important, we can re-prioritize.

[Bobgy]

I think what needs to be done is basically making sure all the local dirs v2 compatible mode launcher reads from/writes to should be accessible by all non-root users.

@Nagarajj may I confirm do you require all containers to run as non-root? or is it OK for some KFP system containers to be root? e.g. we have a kfp-launcher init container that copies the launcher binary to a shared emptyDir volume. Do you want it to be non-root too?

[Bobgy]

I'd imagine the easiest solution/workaround is to

• mount another emptyDir volume for /gcs folder, so it's accessible by all users (because as mentioned, it's part of container contract -- we cannot change the /gcs path)
• change path of /minio/xxx and /s3/xxx to the same folder as the emptyDir containing the launcher binary

For best practice, I think we should move the volume with launcher binary to /var/run/kfp, because /var/run is runtime variable data. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s13.html

So here's the design:

• an empty dir volume is mounted at /var/run/kfp (accessible to all users)
  • kfp launcher binary is copied to /var/run/kfp/bin/launcher
  • input/output s3/minio artifacts are downloaded/written to paths like /var/run/kfp/artifact/s3/xxx, /var/run/kfp/artifact/minio/xxx
  • output parameters are written to paths like /var/run/kfp/parameter/xxxx
• an empty dir volume is mounted at /gcs (accessible to all users)
  • input/output gcs artifacts are downloaded/written to paths like /gcs/xxxx

I think these are not very hard to achieve, maybe I can work on this too when I have some time, but welcome anyone who's interested.

[Nagarajj]

I think what needs to be done is basically making sure all the local dirs v2 compatible mode launcher reads from/writes to should be accessible by all non-root users.

@Nagarajj may I confirm do you require all containers to run as non-root? or is it OK for some KFP system containers to be root? e.g. we have a kfp-launcher init container that copies the launcher binary to a shared emptyDir volume. Do you want it to be non-root too?

If we can remove restriction on Component container to be root it will be good. kfp-laucher init container can be root as we control that.

[Bobgy]

Thanks for the clarification! I think my above design works under these assumptions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dandawg]

I'm working for an enterprise client that is very interested in this issue being resolved. They're in a sensitive data industry, and they are a major target for fraudsters. Hence, they have a need for tight security, and they have a policy of not allowing Kubeflow to run containers as root. Resolving this would allow them to use kubeflow pipelines native artifacts, rather than writing their own detached custom outputs.

[juliusvonkohout]

We also need all container to run a non-root. Since v2compatible is deprecated that should be possible @zijianjoy

[casassg]

hey folks, is there any update on this? I'm guessing current state is that due to 2.0 coming along soon V2_COMPATIBLE wont be worked on? There are several companies running containers in rootless mode - Is. #6530 still up for consideration?

[gregsheremeta]

I posted a PR to address this by doing individual EmptyDir mounts for all the affected locations. I had a hard time trying to move the paths (other than /gcs) as suggested by @Bobgy above, but this seems to work fine and is a pretty small change.

[thesuperzapper]

For those watching, there is a more recent discussion about how this affects V2 (the full version) in:

• [backend] Unable to create directory in Minio when using Artifacts: Permission denied #10397