Skip to content

Commit

Permalink
Kubernetes 1.29 (kubeflow#2716)
Browse files Browse the repository at this point in the history 
Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
  • Loading branch information
@juliusvonkohout
juliusvonkohout authored May 16, 2024
1 parent addf254 commit d2c5275
Show file tree
Hide file tree
Showing 2 changed files with 48 additions and 65 deletions.
107 changes: 45 additions & 62 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

<!-- toc -->

- [Overview](#overview)
- [Overview of the Kubeflow Platform](#overview)
- [Kubeflow components versions](#kubeflow-components-versions)
- [Installation](#installation)
* [Prerequisites](#prerequisites)
Expand All @@ -17,11 +17,11 @@

<!-- tocstop -->

## Overview
## Overview of the Kubeflow Platform

This repo is owned by the [Manifests Working Group](https://github.com/kubeflow/community/blob/master/wg-manifests/charter.md).
This repository is owned by the [Manifests Working Group](https://github.com/kubeflow/community/blob/master/wg-manifests/charter.md).
If you are a contributor authoring or editing the packages please see [Best Practices](./docs/KustomizeBestPractices.md).
Our Slack channel is wg-manifests which you can join here https://www.kubeflow.org/docs/about/community/. You can also find our biweekly meetings there as well.
Our Slack channel is wg-manifests which you can join here https://www.kubeflow.org/docs/about/community/. You can also find there our [biweekly meetings](https://bit.ly/kf-wg-manifests-meet), including the commentable [Agenda](https://bit.ly/kf-wg-manifests-notes)

The Kubeflow Manifests repository is organized under three main directories, which include manifests for installing:

Expand All @@ -31,7 +31,7 @@ The Kubeflow Manifests repository is organized under three main directories, whi
| `common` | Common services, as maintained by the Manifests WG |
| `contrib` | 3rd party contributed applications (e.g. Ray, Kserve), which are maintained externally and are not part of a Kubeflow WG |

All components are deployable with `kustomize`. Any automation tooling for deployment on top of the manifests should be maintained externally by distribution owners.
All components are deployable with `kustomize`. You can choose to deploy the whole Kubeflow platform or individual components.

## Kubeflow components versions

Expand Down Expand Up @@ -82,11 +82,11 @@ The `example` directory contains an example kustomization for the single command
:warning: In both options, we use a default email (`user@example.com`) and password (`12341234`). For any production Kubeflow deployment, you should change the default password by following [the relevant section](#change-default-user-password).

### Prerequisites

- `Kubernetes` (around `1.28`) with a default [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/)
- `kustomize` [5.2.1+](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.2.1)
- :warning: Kubeflow is not compatible with earlier versions of Kustomize. One of the reasons is that we need the [`sortOptions`](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/sortoptions/) field, which is only available in Kustomize 5 and onwards https://github.com/kubeflow/manifests/issues/2388.
- `kubectl`
- This is the master branch which targets Kubernetes 1.29+
- For the specific Kubernetes version per release consult the [release notes](https://github.com/kubeflow/manifests/releases)
- Either our local Kind (installed below) or your own Kubernetes cluster with a default [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/)
- Kustomize [5.2.1+](https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.2.1)
- Kubectl in a version that is [compatible with your Kubernetes cluster](https://kubernetes.io/releases/version-skew-policy/#kubectl)

---
**NOTE**
Expand All @@ -113,7 +113,7 @@ kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.28.0
image: kindest/node:v1.29.4
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
Expand Down Expand Up @@ -142,7 +142,7 @@ kubectl create secret generic regcred \
You can install all Kubeflow official components (residing under `apps`) and all common services (residing under `common`) using the following command:

```sh
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done
```

Once, everything is installed successfully, you can access the Kubeflow Central Dashboard [by logging in to your cluster](#connect-to-your-kubeflow-cluster).
Expand Down Expand Up @@ -176,15 +176,16 @@ If you bump into this error we advise to re-apply the kustomization of the compo

#### cert-manager

cert-manager is used by many Kubeflow components to provide certificates for
Cert-manager is used by many Kubeflow components to provide certificates for
admission webhooks.

Install cert-manager:

```sh
kustomize build common/cert-manager/cert-manager/base | kubectl apply -f -
echo "Waiting for cert-manager to be ready ..."
kubectl wait --for=condition=ready pod -l 'app in (cert-manager,webhook)' --timeout=180s -n cert-manager
kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -
kubectl wait --for=jsonpath='{.subsets[0].addresses[0].targetRef.kind}'=Pod endpoints -l 'app in (cert-manager,webhook)' --timeout=180s -n cert-manager
```

In case you get this error:
Expand All @@ -197,35 +198,33 @@ For more troubleshooting info also check out https://cert-manager.io/docs/troubl

#### Istio

Istio is used by many Kubeflow components to secure their traffic, enforce
Istio is used by most Kubeflow components to secure their traffic, enforce
network authorization and implement routing policies.

Install Istio:

```sh
echo "Installing Istio configured with external authorization..."
cd common/istio-1-17
kustomize build common/istio-1-17/istio-crds/base | kubectl apply -f -
kustomize build common/istio-1-17/istio-namespace/base | kubectl apply -f -
kustomize build common/istio-1-17/istio-install/base | kubectl apply -f -
```

#### AuthService
kustomize build common/istio-1-17/istio-install/overlays/oauth2-proxy | kubectl apply -f -

The OIDC AuthService extends your Istio Ingress-Gateway capabilities, to be able to function as an OIDC client:

```sh
kustomize build common/oidc-client/oidc-authservice/base | kubectl apply -f -
echo "Waiting for all Istio Pods to become ready..."
kubectl wait --for=condition=Ready pods --all -n istio-system --timeout 300s
```

<details>
<summary>oauth2-proxy alternative</summary>
#### Oauth2-proxy

You can use [OAuth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy) instead of [OIDC AuthService](https://github.com/arrikto/oidc-authservice). To do so, run the following command instead
The oauth2-proxy extends your Istio Ingress-Gateway capabilities, to be able to function as an OIDC client:

```sh
kustomize build common/oidc-client/oauth2-proxy/base | kubectl apply -f -
echo "Installing oauth2-proxy..."
kustomize build common/oidc-client/oauth2-proxy/overlays/m2m-self-signed/ | kubectl apply -f -
kubectl wait --for=condition=ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy
```

</details>
It supports user sessions as well as proper token-based machine to machine atuhhentication.

#### Dex

Expand All @@ -234,10 +233,8 @@ Dex is an OpenID Connect Identity (OIDC) with multiple authentication backends.
Install Dex:

```sh
kustomize build common/dex/overlays/istio | kubectl apply -f -
kustomize build common/dex/overlays/oauth2-proxy | kubectl apply -f -
```

> If you are using `oauth2-proxy` as auth envoy filter, you should be using `common/dex/overlays/oauth2-proxy` instead.

#### Knative

Expand Down Expand Up @@ -279,38 +276,19 @@ Install kubeflow roles:
kustomize build common/kubeflow-roles/base | kubectl apply -f -
```

#### Kubeflow Istio Resources

Create the Istio resources needed by Kubeflow. This kustomization currently
creates an Istio Gateway named `kubeflow-gateway`, in namespace `kubeflow`.
If you want to install with your own Istio, then you need this kustomization as
well.

Install istio resources:

```sh
kustomize build common/istio-1-17/kubeflow-istio-resources/base | kubectl apply -f -
```

#### Kubeflow Pipelines

Install the [Multi-User Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/multi-user/) official Kubeflow component:

```sh
kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f -
```
This installs argo with the safe-to use runasnonroot emissary executor. Please note that the installer is still responsible to analyze the security issues that arise when containers are run with root access and to decide if the kubeflow pipeline main containers are run as runasnonroot. It is strongly recommended that the pipelines main containers are installed and run as runasnonroot and without any special capabilities to mitigate security risks.

Do not use the deprecated and insecure PNS executor anymore
```sh
kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user-pns | kubectl apply -f -
```

Refer to [argo workflow executor documentation](https://argoproj.github.io/argo-workflows/workflow-executors) for further reasoning.
This installs argo with the runasnonroot emissary executor. Please note that you are still responsible to analyze the security issues that arise when containers are run with root access and to decide if the kubeflow pipeline main containers are run as runasnonroot. It is in general strongly recommended that all user-accessible OCI containers run with Pod Security Standards [restricted]
(https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted)

**Multi-User Kubeflow Pipelines dependencies**

* Istio + Kubeflow Istio Resources
* Istio
* Kubeflow Roles
* OIDC Auth Service (or cloud provider specific auth service)
* Profiles + KFAM
Expand All @@ -337,14 +315,12 @@ Install the KServe component:
kustomize build contrib/kserve/kserve | kubectl apply -f -
```

Install the Models web app:
Install the Models web application:

```sh
kustomize build contrib/kserve/models-web-app/overlays/kubeflow | kubectl apply -f -
```

- ../contrib/kserve/models-web-app/overlays/kubeflow

#### Katib

Install the Katib official Kubeflow component:
Expand All @@ -369,7 +345,7 @@ Install the Admission Webhook for PodDefaults:
kustomize build apps/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -
```

#### Notebooks
#### Notebooks 1.0

Install the Notebook Controller official Kubeflow component:

Expand All @@ -383,6 +359,10 @@ Install the Jupyter Web App official Kubeflow component:
kustomize build apps/jupyter/jupyter-web-app/upstream/overlays/istio | kubectl apply -f -
```

#### Workspaces (Notebooks 2.0)

It is still in development.

#### PVC Viewer Controller

Install the PVC Viewer Controller official Kubeflow component:
Expand All @@ -400,7 +380,7 @@ components:
kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -
```

#### Volumes Web App
#### Volumes Web Application

Install the Volumes Web App official Kubeflow component:

Expand Down Expand Up @@ -430,7 +410,7 @@ Install the Training Operator official Kubeflow component:
kustomize build apps/training-operator/upstream/overlays/kubeflow | kubectl apply -f -
```

#### User Namespace
#### User Namespaces

Finally, create a new namespace for the default user (named `kubeflow-user-example-com`).

Expand Down Expand Up @@ -463,13 +443,14 @@ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
After running the command, you can access the Kubeflow Central Dashboard by doing the following:

1. Open your browser and visit `http://localhost:8080`. You should get the Dex login screen.
2. Login with the default user's credential. The default email address is `user@example.com` and the default password is `12341234`.
2. Login with the default user's credentials. The default email address is `user@example.com` and the default password is `12341234`.

#### NodePort / LoadBalancer / Ingress

In order to connect to Kubeflow using NodePort / LoadBalancer / Ingress, you need to setup HTTPS. The reason is that many of our web apps (e.g., Tensorboard Web App, Jupyter Web App, Katib UI) use [Secure Cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies), so accessing Kubeflow with HTTP over a non-localhost domain does not work.
In order to connect to Kubeflow using NodePort / LoadBalancer / Ingress, you need to setup HTTPS. The reason is that many of our web applications (e.g., Tensorboard Web Application, Jupyter Web Application, Katib UI) use [Secure Cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies), so accessing Kubeflow with HTTP over a non-localhost domain does not work.

Exposing your Kubeflow cluster with proper HTTPS is a process heavily dependent on your environment. For this reason, please take a look at the available [Kubeflow distributions](https://www.kubeflow.org/docs/started/installing-kubeflow/#install-a-packaged-kubeflow-distribution), which are targeted to specific environments, and select the one that fits your needs.
Exposing your Kubeflow cluster with proper HTTPS is a simple proces, but dependent on your environment.
There are also third-party commercial [distributions](https://www.kubeflow.org/docs/started/installing-kubeflow/#install-a-packaged-kubeflow-distribution) available.

---
**NOTE**
Expand All @@ -484,6 +465,8 @@ For security reasons, we don't want to use the default password for the default

1. Pick a password for the default user, with email `user@example.com`, and hash it using `bcrypt`:

TODO this changed slightly in https://github.com/kubeflow/manifests/pull/2669 and https://github.com/kubeflow/manifests/pull/2229

```sh
python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))'
```
Expand Down
6 changes: 3 additions & 3 deletions tests/gh-actions/kind-cluster.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ kubeadmConfigPatches:
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
nodes:
- role: control-plane
image: kindest/node:v1.28.7@sha256:a99353a19f3f8958bc4d9b808425224b6ff0f03b0c1ef7ae18eda5ccc0c21342
image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8
- role: worker
image: kindest/node:v1.28.7@sha256:a99353a19f3f8958bc4d9b808425224b6ff0f03b0c1ef7ae18eda5ccc0c21342
image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8
- role: worker
image: kindest/node:v1.28.7@sha256:a99353a19f3f8958bc4d9b808425224b6ff0f03b0c1ef7ae18eda5ccc0c21342
image: kindest/node:v1.29.4@sha256:3abb816a5b1061fb15c6e9e60856ec40d56b7b52bcea5f5f1350bc6e2320b6f8

0 comments on commit d2c5275

Please sign in to comment.