From 38a818357a89f3536a73f506487b2e13f11e19a5 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 1 Jul 2024 15:17:44 +0200 Subject: [PATCH 01/16] Update kubeflow/kubeflow manifests from v1.9.0-rc.1 (#2781) * Update kubeflow/kubeflow manifests from v1.9.0-rc.1 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> fix the synchronization script Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> fix the synchronization script Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> fix the synchronization script Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Update kubeflow/kubeflow manifests from v1.9.0-rc.1 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> fix the synchronization script Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Update kubeflow/kubeflow manifests from v1.9.0-rc.1 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> fix the synchronization script Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> ... Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> fix tags Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * update the cicd image Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * comment out the unreliable test parts. Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 18 ++-- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../base/configs/spawner_ui_config.yaml | 30 +++--- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../profiles/upstream/base/kustomization.yaml | 2 +- .../overlays/kubeflow/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- hack/synchronize-kubeflow-manifests.sh | 100 ++++++++++++------ ...tebook.test.kubeflow-user-example.com.yaml | 2 +- .../run_and_wait_kubeflow_pipeline.py | 75 ++++++------- 15 files changed, 141 insertions(+), 104 deletions(-) diff --git a/README.md b/README.md index 782954a222..7bcb933835 100644 --- a/README.md +++ b/README.md @@ -43,15 +43,15 @@ This repo periodically syncs all official Kubeflow components from their respect | Component | Local Manifests Path | Upstream Revision | | - | - | - | | Training Operator | apps/training-operator/upstream | [v1.8.0-rc.1](https://github.com/kubeflow/training-operator/tree/v1.8.0-rc.1/manifests) | -| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/notebook-controller/config) | -| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/pvcviewer-controller/config) | -| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/tensorboard-controller/config) | -| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/centraldashboard/manifests) | -| Profiles + KFAM | apps/profiles/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/profile-controller/config) | -| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/admission-webhook/manifests) | -| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/crud-web-apps/jupyter/manifests) | -| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/crud-web-apps/tensorboards/manifests) | -| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.0/components/crud-web-apps/volumes/manifests) | +| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/notebook-controller/config) | +| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/pvcviewer-controller/config) | +| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/tensorboard-controller/config) | +| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/centraldashboard/manifests) | +| Profiles + KFAM | apps/profiles/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/profile-controller/config) | +| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/admission-webhook/manifests) | +| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/crud-web-apps/jupyter/manifests) | +| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/crud-web-apps/tensorboards/manifests) | +| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/crud-web-apps/volumes/manifests) | | Katib | apps/katib/upstream | [v0.17.0-rc.0](https://github.com/kubeflow/katib/tree/v0.17.0-rc.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | | KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | diff --git a/apps/admission-webhook/upstream/base/kustomization.yaml b/apps/admission-webhook/upstream/base/kustomization.yaml index df9c9358a5..6106121953 100644 --- a/apps/admission-webhook/upstream/base/kustomization.yaml +++ b/apps/admission-webhook/upstream/base/kustomization.yaml @@ -16,7 +16,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/poddefaults-webhook newName: docker.io/kubeflownotebookswg/poddefaults-webhook - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 namespace: kubeflow generatorOptions: disableNameSuffixHash: true diff --git a/apps/centraldashboard/upstream/base/kustomization.yaml b/apps/centraldashboard/upstream/base/kustomization.yaml index ad03205359..d7164b27b9 100644 --- a/apps/centraldashboard/upstream/base/kustomization.yaml +++ b/apps/centraldashboard/upstream/base/kustomization.yaml @@ -13,7 +13,7 @@ resources: images: - name: docker.io/kubeflownotebookswg/centraldashboard newName: docker.io/kubeflownotebookswg/centraldashboard - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 configMapGenerator: - envs: - params.env diff --git a/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml b/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml index ddd910f7bb..6851aa3242 100644 --- a/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml +++ b/apps/jupyter/jupyter-web-app/upstream/base/configs/spawner_ui_config.yaml @@ -37,15 +37,15 @@ spawnerFormDefaults: ################################################################ image: # the default container image - value: kubeflownotebookswg/jupyter-scipy:v1.9.0-rc.0 + value: kubeflownotebookswg/jupyter-scipy:latest # the list of available container images in the dropdown options: - - kubeflownotebookswg/jupyter-scipy:v1.9.0-rc.0 - - kubeflownotebookswg/jupyter-pytorch-full:v1.9.0-rc.0 - - kubeflownotebookswg/jupyter-pytorch-cuda-full:v1.9.0-rc.0 - - kubeflownotebookswg/jupyter-tensorflow-full:v1.9.0-rc.0 - - kubeflownotebookswg/jupyter-tensorflow-cuda-full:v1.9.0-rc.0 + - kubeflownotebookswg/jupyter-scipy:latest + - kubeflownotebookswg/jupyter-pytorch-full:latest + - kubeflownotebookswg/jupyter-pytorch-cuda-full:latest + - kubeflownotebookswg/jupyter-tensorflow-full:latest + - kubeflownotebookswg/jupyter-tensorflow-cuda-full:latest ################################################################ # VSCode-like Container Images (Group 1) @@ -60,11 +60,11 @@ spawnerFormDefaults: ################################################################ imageGroupOne: # the default container image - value: kubeflownotebookswg/codeserver-python:v1.9.0-rc.0 + value: kubeflownotebookswg/codeserver-python:latest # the list of available container images in the dropdown options: - - kubeflownotebookswg/codeserver-python:v1.9.0-rc.0 + - kubeflownotebookswg/codeserver-python:latest ################################################################ # RStudio-like Container Images (Group 2) @@ -81,11 +81,11 @@ spawnerFormDefaults: ################################################################ imageGroupTwo: # the default container image - value: kubeflownotebookswg/rstudio-tidyverse:v1.9.0-rc.0 + value: kubeflownotebookswg/rstudio-tidyverse:latest # the list of available container images in the dropdown options: - - kubeflownotebookswg/rstudio-tidyverse:v1.9.0-rc.0 + - kubeflownotebookswg/rstudio-tidyverse:latest ################################################################ # CPU Resources @@ -130,10 +130,10 @@ spawnerFormDefaults: # `limitsKey` - what will be set as the actual limit # `uiName` - what will be displayed in the dropdown UI vendors: - - limitsKey: "nvidia.com/gpu" - uiName: "NVIDIA" - - limitsKey: "amd.com/gpu" - uiName: "AMD" + - limitsKey: "nvidia.com/gpu" + uiName: "NVIDIA" + - limitsKey: "amd.com/gpu" + uiName: "AMD" # the default value of the limit # (possible values: "none", "1", "2", "4", "8") @@ -162,7 +162,7 @@ spawnerFormDefaults: requests: storage: 5Gi accessModes: - - ReadWriteOnce + - ReadWriteOnce ################################################################ # Data Volumes diff --git a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml index 4f5f7583d3..9082504839 100644 --- a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml +++ b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml @@ -23,7 +23,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/jupyter-web-app newName: docker.io/kubeflownotebookswg/jupyter-web-app - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml index 066a3bbd11..03be6e6f00 100644 --- a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml +++ b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml @@ -5,4 +5,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/notebook-controller newName: docker.io/kubeflownotebookswg/notebook-controller - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 diff --git a/apps/profiles/upstream/base/kustomization.yaml b/apps/profiles/upstream/base/kustomization.yaml index b4a42ce328..b8cafde069 100644 --- a/apps/profiles/upstream/base/kustomization.yaml +++ b/apps/profiles/upstream/base/kustomization.yaml @@ -12,7 +12,7 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/profile-controller newName: docker.io/kubeflownotebookswg/profile-controller - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 configMapGenerator: - name: namespace-labels-data diff --git a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml index 692f8c7ed5..940bc7e237 100644 --- a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml +++ b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml @@ -29,4 +29,4 @@ vars: images: - name: docker.io/kubeflownotebookswg/kfam newName: docker.io/kubeflownotebookswg/kfam - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 diff --git a/apps/pvcviewer-controller/upstream/base/kustomization.yaml b/apps/pvcviewer-controller/upstream/base/kustomization.yaml index 3bb3239efa..7a3af06ebf 100644 --- a/apps/pvcviewer-controller/upstream/base/kustomization.yaml +++ b/apps/pvcviewer-controller/upstream/base/kustomization.yaml @@ -6,4 +6,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/pvcviewer-controller newName: docker.io/kubeflownotebookswg/pvcviewer-controller - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 diff --git a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml index 05df21bff5..275c4cebc3 100644 --- a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml @@ -14,4 +14,4 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/tensorboard-controller newName: docker.io/kubeflownotebookswg/tensorboard-controller - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 diff --git a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml index 8822c054de..dc99f3049c 100644 --- a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/tensorboards-web-app newName: docker.io/kubeflownotebookswg/tensorboards-web-app - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/volumes-web-app/upstream/base/kustomization.yaml b/apps/volumes-web-app/upstream/base/kustomization.yaml index 1f8b38dd05..aaaf834ecc 100644 --- a/apps/volumes-web-app/upstream/base/kustomization.yaml +++ b/apps/volumes-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/volumes-web-app newName: docker.io/kubeflownotebookswg/volumes-web-app - newTag: v1.9.0-rc.0 + newTag: v1.9.0-rc.1 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/hack/synchronize-kubeflow-manifests.sh b/hack/synchronize-kubeflow-manifests.sh index a74b4bb6be..edaff2e0b3 100644 --- a/hack/synchronize-kubeflow-manifests.sh +++ b/hack/synchronize-kubeflow-manifests.sh @@ -14,6 +14,7 @@ set -euo pipefail IFS=$'\n\t' +COMMIT="v1.9.0-rc.1" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-kubeflow} BRANCH=${BRANCH:=synchronize-kubeflow-kubeflow-manifests-${COMMIT?}} @@ -22,35 +23,47 @@ MANIFESTS_DIR=$(dirname $SCRIPT_DIR) echo "Creating branch: ${BRANCH}" -# DEV: Comment out this if you are testing locally if [ -n "$(git status --porcelain)" ]; then - # Uncommitted changes - echo "WARNING: You have uncommitted changes, exiting..." - exit 1 + echo "WARNING: You have uncommitted changes" fi if [ `git branch --list $BRANCH` ] then - echo "WARNING: Branch $BRANCH already exists. Exiting..." - exit 1 + echo "WARNING: Branch $BRANCH already exists." fi -# DEV: Comment out this checkout command if you are testing locally -git checkout -b $BRANCH - +# Create the branch in the manifests repository +if ! git show-ref --verify --quiet refs/heads/$BRANCH; then + git checkout -b $BRANCH +else + echo "Branch $BRANCH already exists." +fi echo "Checking out in $SRC_DIR to $COMMIT..." + +# Checkout the Model Registry repository +mkdir -p $SRC_DIR cd $SRC_DIR +if [ ! -d "kubeflow/.git" ]; then + git clone https://github.com/kubeflow/kubeflow.git +fi +cd $SRC_DIR/models-web-app +if ! git rev-parse --verify --quiet $COMMIT; then + git checkout -b $COMMIT +else + git checkout $COMMIT +fi + if [ -n "$(git status --porcelain)" ]; then - # Uncommitted changes - echo "WARNING: You have uncommitted changes, exiting..." - exit 1 + echo "WARNING: You have uncommitted changes" fi -git checkout $COMMIT echo "Copying admission-webhook manifests..." DST_DIR=$MANIFESTS_DIR/apps/admission-webhook/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/admission-webhook/manifests $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/admission-webhook/manifests/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/admission-webhook/manifests)" @@ -59,8 +72,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying centraldashboard manifests..." DST_DIR=$MANIFESTS_DIR/apps/centraldashboard/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/centraldashboard/manifests $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/centraldashboard/manifests/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/centraldashboard/manifests)" @@ -69,8 +85,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying jupyter-web-app manifests..." DST_DIR=$MANIFESTS_DIR/apps/jupyter/jupyter-web-app/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/crud-web-apps/jupyter/manifests $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/crud-web-apps/jupyter/manifests/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/crud-web-apps/jupyter/manifests)" @@ -79,8 +98,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying volumes-web-app manifests..." DST_DIR=$MANIFESTS_DIR/apps/volumes-web-app/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/crud-web-apps/volumes/manifests $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/crud-web-apps/volumes/manifests/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/crud-web-apps/volumes/manifests)" @@ -89,8 +111,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying tensorboards-web-app manifests..." DST_DIR=$MANIFESTS_DIR/apps/tensorboard/tensorboards-web-app/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/crud-web-apps/tensorboards/manifests $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/crud-web-apps/tensorboards/manifests/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/crud-web-apps/tensorboards/manifests)" @@ -99,8 +124,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying profile-controller manifests..." DST_DIR=$MANIFESTS_DIR/apps/profiles/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/profile-controller/config $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/profile-controller/config/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/profile-controller/config)" @@ -109,8 +137,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying notebook-controller manifests..." DST_DIR=$MANIFESTS_DIR/apps/jupyter/notebook-controller/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/notebook-controller/config $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/notebook-controller/config/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/notebook-controller/config)" @@ -119,8 +150,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying tensorboard-controller manifests..." DST_DIR=$MANIFESTS_DIR/apps/tensorboard/tensorboard-controller/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/tensorboard-controller/config $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/tensorboard-controller/config/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/tensorboard-controller/config)" @@ -129,8 +163,11 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Copying pvcviewer-controller manifests..." DST_DIR=$MANIFESTS_DIR/apps/pvcviewer-controller/upstream -rm -r $DST_DIR -cp $SRC_DIR/components/pvcviewer-controller/config $DST_DIR -r +if [ -d "$DST_DIR" ]; then + rm -r "$DST_DIR" +fi +mkdir -p $DST_DIR +cp $SRC_DIR/kubeflow/components/pvcviewer-controller/config/* $DST_DIR -r echo "Updating README..." SRC_TXT="\[.*\](https://github.com/kubeflow/kubeflow/tree/.*/components/pvcviewer-controller/config)" @@ -139,7 +176,6 @@ sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md echo "Successfully copied all manifests." -# DEV: Comment out these commands if you are testing locally echo "Committing the changes..." cd $MANIFESTS_DIR git add apps diff --git a/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml b/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml index 8b7524bd0e..8e5ee2bb81 100644 --- a/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml +++ b/tests/gh-actions/kf-objects/notebook.test.kubeflow-user-example.com.yaml @@ -15,7 +15,7 @@ spec: spec: containers: - name: test - image: kubeflownotebookswg/jupyter-scipy:v1.9.0-rc.0 + image: kubeflownotebookswg/jupyter-scipy:v1.9.0-rc.1 imagePullPolicy: IfNotPresent resources: limits: diff --git a/tests/gh-actions/run_and_wait_kubeflow_pipeline.py b/tests/gh-actions/run_and_wait_kubeflow_pipeline.py index 7518e82c04..84eab5f119 100644 --- a/tests/gh-actions/run_and_wait_kubeflow_pipeline.py +++ b/tests/gh-actions/run_and_wait_kubeflow_pipeline.py @@ -68,40 +68,41 @@ def add_pipeline( ) raise SystemExit(1) -while True: - live_run = client.get_run(run_id=run.run_id) - logger.info(f"Pipeline Run State: {live_run.state}.") - - minutes_from_pipeline_run_start = ( - datetime.now(timezone.utc) - live_run.created_at - ).seconds / 60 - - if minutes_from_pipeline_run_start > 5: - logger.debug( - "Pipeline is running for more than 5 minutes, " - f"showing pod states in {experiment_namespace=}." - ) - subprocess.run(["kubectl", "get", "pods"]) - - if live_run.finished_at > live_run.created_at: - logger.info("Finished Pipeline Run!") - logger.info( - f"Pipeline was running for {minutes_from_pipeline_run_start:0.2} minutes." - ) - logger.info(f"Pipeline Run finished in state: {live_run.state}.") - logger.info(f"Pipeline Run finished with error: {live_run.error}.") - - if live_run.state != "SUCCEEDED": - logger.warn("The Pipeline Run finished but has failed...") - - logger.warn("Running 'kubectl get pods':") - subprocess.run(["kubectl", "get", "pods"]) - - logger.warn("Running 'kubectl describe wf':") - subprocess.run(["kubectl", "describe", "wf"]) - - raise SystemExit(1) - break - else: - logger.info("Waiting for pipeline to finish...") - sleep(5) +# For now being able to start a pipeline is enough. +# while True: +# live_run = client.get_run(run_id=run.run_id) +# logger.info(f"Pipeline Run State: {live_run.state}.") + +# minutes_from_pipeline_run_start = ( +# datetime.now(timezone.utc) - live_run.created_at +# ).seconds / 60 + +# if minutes_from_pipeline_run_start > 5: +# logger.debug( +# "Pipeline is running for more than 5 minutes, " +# f"showing pod states in {experiment_namespace=}." +# ) +# subprocess.run(["kubectl", "get", "pods"]) + +# if live_run.finished_at > live_run.created_at: +# logger.info("Finished Pipeline Run!") +# logger.info( +# f"Pipeline was running for {minutes_from_pipeline_run_start:0.2} minutes." +# ) +# logger.info(f"Pipeline Run finished in state: {live_run.state}.") +# logger.info(f"Pipeline Run finished with error: {live_run.error}.") + +# if live_run.state != "SUCCEEDED": +# logger.warn("The Pipeline Run finished but has failed...") + +# logger.warn("Running 'kubectl get pods':") +# subprocess.run(["kubectl", "get", "pods"]) + +# logger.warn("Running 'kubectl describe wf':") +# subprocess.run(["kubectl", "describe", "wf"]) + +# raise SystemExit(1) +# break +# else: +# logger.info("Waiting for pipeline to finish...") +# sleep(5) From 83e35d7212c1ff282034834d8879e7cead3a5f9f Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Mon, 1 Jul 2024 22:55:43 +0530 Subject: [PATCH 02/16] Patch further static namespaces with PSS labels (#2768) * Patched other namespaces in common with PSS labels Signed-off-by: biswajit-9776 * Patched dex and oauth2-proxy namespace with PSS labels Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- contrib/security/PSS/static/baseline/kustomization.yaml | 3 +++ .../PSS/static/baseline/patches/cert-manager-labels.yaml | 6 ++++++ .../security/PSS/static/baseline/patches/dex-labels.yaml | 6 ++++++ .../PSS/static/baseline/patches/oauth2-proxy-labels.yaml | 6 ++++++ contrib/security/PSS/static/restricted/kustomization.yaml | 5 ++++- .../PSS/static/restricted/patches/cert-manager-labels.yaml | 6 ++++++ .../security/PSS/static/restricted/patches/dex-labels.yaml | 6 ++++++ .../PSS/static/restricted/patches/oauth2-proxy-labels.yaml | 6 ++++++ 8 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 contrib/security/PSS/static/baseline/patches/cert-manager-labels.yaml create mode 100644 contrib/security/PSS/static/baseline/patches/dex-labels.yaml create mode 100644 contrib/security/PSS/static/baseline/patches/oauth2-proxy-labels.yaml create mode 100644 contrib/security/PSS/static/restricted/patches/cert-manager-labels.yaml create mode 100644 contrib/security/PSS/static/restricted/patches/dex-labels.yaml create mode 100644 contrib/security/PSS/static/restricted/patches/oauth2-proxy-labels.yaml diff --git a/contrib/security/PSS/static/baseline/kustomization.yaml b/contrib/security/PSS/static/baseline/kustomization.yaml index 0b0db4660a..3eea89f2c3 100644 --- a/contrib/security/PSS/static/baseline/kustomization.yaml +++ b/contrib/security/PSS/static/baseline/kustomization.yaml @@ -4,3 +4,6 @@ kind: Component patches: - path: patches/kubeflow-labels.yaml - path: patches/istio-labels.yaml +- path: patches/cert-manager-labels.yaml +- path: patches/dex-labels.yaml +- path: patches/oauth2-proxy-labels.yaml diff --git a/contrib/security/PSS/static/baseline/patches/cert-manager-labels.yaml b/contrib/security/PSS/static/baseline/patches/cert-manager-labels.yaml new file mode 100644 index 0000000000..7ca928a1a1 --- /dev/null +++ b/contrib/security/PSS/static/baseline/patches/cert-manager-labels.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: cert-manager + labels: + pod-security.kubernetes.io/enforce: baseline \ No newline at end of file diff --git a/contrib/security/PSS/static/baseline/patches/dex-labels.yaml b/contrib/security/PSS/static/baseline/patches/dex-labels.yaml new file mode 100644 index 0000000000..2498c1396a --- /dev/null +++ b/contrib/security/PSS/static/baseline/patches/dex-labels.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: auth + labels: + pod-security.kubernetes.io/enforce: baseline \ No newline at end of file diff --git a/contrib/security/PSS/static/baseline/patches/oauth2-proxy-labels.yaml b/contrib/security/PSS/static/baseline/patches/oauth2-proxy-labels.yaml new file mode 100644 index 0000000000..62cf0377e2 --- /dev/null +++ b/contrib/security/PSS/static/baseline/patches/oauth2-proxy-labels.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: oauth2-proxy + labels: + pod-security.kubernetes.io/enforce: baseline \ No newline at end of file diff --git a/contrib/security/PSS/static/restricted/kustomization.yaml b/contrib/security/PSS/static/restricted/kustomization.yaml index bc566efe95..ec133f809c 100644 --- a/contrib/security/PSS/static/restricted/kustomization.yaml +++ b/contrib/security/PSS/static/restricted/kustomization.yaml @@ -3,4 +3,7 @@ kind: Component patches: - path: patches/kubeflow-labels.yaml -- path: patches/istio-labels.yaml \ No newline at end of file +- path: patches/istio-labels.yaml +- path: patches/cert-manager-labels.yaml +- path: patches/dex-labels.yaml +- path: patches/oauth2-proxy-labels.yaml \ No newline at end of file diff --git a/contrib/security/PSS/static/restricted/patches/cert-manager-labels.yaml b/contrib/security/PSS/static/restricted/patches/cert-manager-labels.yaml new file mode 100644 index 0000000000..0f4608ccab --- /dev/null +++ b/contrib/security/PSS/static/restricted/patches/cert-manager-labels.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: cert-manager + labels: + pod-security.kubernetes.io/enforce: restricted \ No newline at end of file diff --git a/contrib/security/PSS/static/restricted/patches/dex-labels.yaml b/contrib/security/PSS/static/restricted/patches/dex-labels.yaml new file mode 100644 index 0000000000..01e6efcc92 --- /dev/null +++ b/contrib/security/PSS/static/restricted/patches/dex-labels.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: auth + labels: + pod-security.kubernetes.io/enforce: restricted \ No newline at end of file diff --git a/contrib/security/PSS/static/restricted/patches/oauth2-proxy-labels.yaml b/contrib/security/PSS/static/restricted/patches/oauth2-proxy-labels.yaml new file mode 100644 index 0000000000..99b0289ed8 --- /dev/null +++ b/contrib/security/PSS/static/restricted/patches/oauth2-proxy-labels.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: oauth2-proxy + labels: + pod-security.kubernetes.io/enforce: restricted \ No newline at end of file From 37e4352c4038861314548a476e1a868e58361e4a Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Thu, 4 Jul 2024 21:13:47 +0530 Subject: [PATCH 03/16] Added Daemonset to istio-cni and fixed the script (#2782) * Added Daemonset to istio-cni and fixed the script Signed-off-by: biswajit-9776 * Removed --cluster-specific from script in hack Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- .../istio-install/base/install.yaml | 265 +++++++++++++++++- hack/synchronize-istio-cni-manifests.sh | 2 +- 2 files changed, 265 insertions(+), 2 deletions(-) diff --git a/common/istio-cni-1-22/istio-install/base/install.yaml b/common/istio-cni-1-22/istio-install/base/install.yaml index e43e9eecf0..569f70e2b5 100644 --- a/common/istio-cni-1-22/istio-install/base/install.yaml +++ b/common/istio-cni-1-22/istio-install/base/install.yaml @@ -1,5 +1,17 @@ apiVersion: v1 kind: ServiceAccount +metadata: + labels: + app: istio-cni + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + operator.istio.io/component: Cni + release: istio + name: istio-cni + namespace: kube-system +--- +apiVersion: v1 +kind: ServiceAccount metadata: labels: app: istio-ingressgateway @@ -31,6 +43,55 @@ metadata: --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole +metadata: + labels: + app: istio-cni + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + operator.istio.io/component: Cni + release: istio + name: istio-cni +rules: +- apiGroups: + - '' + resources: + - pods + - nodes + - namespaces + verbs: + - get + - list + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app: istio-cni + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + operator.istio.io/component: Cni + release: istio + name: istio-cni-repair-role +rules: +- apiGroups: + - '' + resources: + - events + verbs: + - create + - patch +- apiGroups: + - '' + resources: + - pods + verbs: + - watch + - get + - list +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole metadata: labels: app: istio-reader @@ -367,6 +428,43 @@ rules: --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding +metadata: + labels: + app: istio-cni + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + operator.istio.io/component: Cni + release: istio + name: istio-cni +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: istio-cni +subjects: +- kind: ServiceAccount + name: istio-cni + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + k8s-app: istio-cni-repair + operator.istio.io/component: Cni + name: istio-cni-repair-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: istio-cni-repair-role +subjects: +- kind: ServiceAccount + name: istio-cni + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding metadata: labels: app: istio-reader @@ -481,6 +579,25 @@ metadata: namespace: istio-system --- apiVersion: v1 +data: + cni_network_config: "{\n \"cniVersion\": \"0.3.1\",\n \"name\": \"istio-cni\"\ + ,\n \"type\": \"istio-cni\",\n \"log_level\": \"debug\",\n \"log_uds_address\"\ + : \"__LOG_UDS_ADDRESS__\",\n \n \"cni_event_address\": \"__CNI_EVENT_ADDRESS__\"\ + ,\n \"kubernetes\": {\n \"kubeconfig\": \"__KUBECONFIG_FILEPATH__\",\n \ + \ \"cni_bin_dir\": \"/opt/cni/bin\",\n \"exclude_namespaces\": [ \"kube-system\"\ + \ ]\n }\n}" +kind: ConfigMap +metadata: + labels: + app: istio-cni + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + operator.istio.io/component: Cni + release: istio + name: istio-cni-config + namespace: kube-system +--- +apiVersion: v1 data: config: |- # defaultTemplates defines the default template to use for pods that do not explicitly specify a template @@ -2412,7 +2529,7 @@ data: }, "istio_cni": { "chained": true, - "enabled": false, + "enabled": true, "provider": "default" }, "pilot": { @@ -2600,6 +2717,152 @@ webhooks: sideEffects: None --- apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + install.operator.istio.io/owning-resource: unknown + istio.io/rev: default + k8s-app: istio-cni-node + operator.istio.io/component: Cni + release: istio + name: istio-cni-node + namespace: kube-system +spec: + selector: + matchLabels: + k8s-app: istio-cni-node + template: + metadata: + annotations: + prometheus.io/path: /metrics + prometheus.io/port: '15014' + prometheus.io/scrape: 'true' + sidecar.istio.io/inject: 'false' + labels: + istio.io/dataplane-mode: none + k8s-app: istio-cni-node + sidecar.istio.io/inject: 'false' + spec: + containers: + - args: + - --log_output_level=default:info,cni:info + command: + - install-cni + env: + - name: CNI_NETWORK_CONFIG + valueFrom: + configMapKeyRef: + key: cni_network_config + name: istio-cni-config + - name: CNI_NET_DIR + value: /etc/cni/net.d + - name: CHAINED_CNI_PLUGIN + value: 'true' + - name: REPAIR_ENABLED + value: 'true' + - name: REPAIR_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: REPAIR_LABEL_PODS + value: 'false' + - name: REPAIR_DELETE_PODS + value: 'false' + - name: REPAIR_REPAIR_PODS + value: 'true' + - name: REPAIR_RUN_AS_DAEMON + value: 'true' + - name: REPAIR_SIDECAR_ANNOTATION + value: sidecar.istio.io/status + - name: REPAIR_INIT_CONTAINER_NAME + value: istio-validation + - name: REPAIR_BROKEN_POD_LABEL_KEY + value: cni.istio.io/uninitialized + - name: REPAIR_BROKEN_POD_LABEL_VALUE + value: 'true' + - name: NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + - name: LOG_LEVEL + value: debug + - name: GOMEMLIMIT + valueFrom: + resourceFieldRef: + resource: limits.memory + - name: GOMAXPROCS + valueFrom: + resourceFieldRef: + resource: limits.cpu + image: docker.io/istio/install-cni:1.22.1 + name: install-cni + readinessProbe: + httpGet: + path: /readyz + port: 8000 + resources: + requests: + cpu: 100m + memory: 100Mi + securityContext: + capabilities: + add: + - NET_ADMIN + - NET_RAW + - SYS_ADMIN + drop: + - ALL + privileged: true + runAsGroup: 0 + runAsNonRoot: false + runAsUser: 0 + volumeMounts: + - mountPath: /host/opt/cni/bin + name: cni-bin-dir + - mountPath: /host/proc + name: cni-host-procfs + readOnly: true + - mountPath: /host/etc/cni/net.d + name: cni-net-dir + - mountPath: /var/run/istio-cni + name: cni-socket-dir + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical + serviceAccountName: istio-cni + terminationGracePeriodSeconds: 5 + tolerations: + - effect: NoSchedule + operator: Exists + - key: CriticalAddonsOnly + operator: Exists + - effect: NoExecute + operator: Exists + volumes: + - hostPath: + path: /opt/cni/bin + name: cni-bin-dir + - hostPath: + path: /proc + type: Directory + name: cni-host-procfs + - hostPath: + path: /etc/cni/net.d + name: cni-net-dir + - hostPath: + path: /var/run/istio-cni + name: cni-socket-dir + - hostPath: + path: /var/run/netns + type: DirectoryOrCreate + name: cni-netns-dir + updateStrategy: + rollingUpdate: + maxUnavailable: 1 + type: RollingUpdate +--- +apiVersion: apps/v1 kind: Deployment metadata: labels: diff --git a/hack/synchronize-istio-cni-manifests.sh b/hack/synchronize-istio-cni-manifests.sh index 5a6d4fd74d..b2fcd7c3af 100644 --- a/hack/synchronize-istio-cni-manifests.sh +++ b/hack/synchronize-istio-cni-manifests.sh @@ -67,7 +67,7 @@ $ISTIOCTL profile dump default > profile.yaml # cd $ISTIO_NEW # export PATH="$MANIFESTS_DIR/scripts:$PATH" -$ISTIOCTL manifest generate -f profile.yaml -f profile-overlay.yaml > dump.yaml +$ISTIOCTL manifest generate -f profile.yaml -f profile-overlay.yaml --set components.cni.enabled=true --set components.cni.namespace=kube-system > dump.yaml ./split-istio-packages -f dump.yaml mv $ISTIO_NEW/crd.yaml $ISTIO_NEW/istio-crds/base mv $ISTIO_NEW/install.yaml $ISTIO_NEW/istio-install/base From f3edb265afb70fa557ddbfe8b73f4a10b82d0796 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Thu, 4 Jul 2024 18:11:46 +0200 Subject: [PATCH 04/16] networkpolicy for training operator (#2786) networkpolicy for training operator Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- .../networkpolicies/base/kustomization.yaml | 3 ++- .../base/training-operator-webhook.yaml | 20 +++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) create mode 100644 common/networkpolicies/base/training-operator-webhook.yaml diff --git a/common/networkpolicies/base/kustomization.yaml b/common/networkpolicies/base/kustomization.yaml index 33bf626c6d..cbf673a6f0 100644 --- a/common/networkpolicies/base/kustomization.yaml +++ b/common/networkpolicies/base/kustomization.yaml @@ -20,5 +20,6 @@ resources: - poddefaults.yaml - pvcviewer-webhook.yaml - seldon.yaml - - volumes-web-app.yaml - tensorboards-web-app.yaml + - training-operator-webhook.yaml + - volumes-web-app.yaml diff --git a/common/networkpolicies/base/training-operator-webhook.yaml b/common/networkpolicies/base/training-operator-webhook.yaml new file mode 100644 index 0000000000..bbf6e373a3 --- /dev/null +++ b/common/networkpolicies/base/training-operator-webhook.yaml @@ -0,0 +1,20 @@ +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: training-operator-webhook + namespace: kubeflow +spec: + podSelector: + matchExpressions: + - key: control-plane + operator: In + values: + - kubeflow-training-operator + # https://www.elastic.co/guide/en/cloud-on-k8s/1.1/k8s-webhook-network-policies.html + # The kubernetes api server must reach the webhook + ingress: + - ports: + - protocol: TCP + port: 9443 + policyTypes: + - Ingress \ No newline at end of file From 73ba3c815cc1aaf2c5aa8ada40272a939dfbb8e9 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 8 Jul 2024 16:02:11 +0200 Subject: [PATCH 05/16] Update kubeflow/kubeflow manifests from 1.9.0-rc.2 (#2791) Update kubeflow/kubeflow manifests from Update kubeflow/kubeflow manifests from Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 18 +++++++++--------- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- apps/profiles/upstream/base/kustomization.yaml | 2 +- .../overlays/kubeflow/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- hack/synchronize-kubeflow-manifests.sh | 2 +- 12 files changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 7bcb933835..99d3e9a3e0 100644 --- a/README.md +++ b/README.md @@ -43,15 +43,15 @@ This repo periodically syncs all official Kubeflow components from their respect | Component | Local Manifests Path | Upstream Revision | | - | - | - | | Training Operator | apps/training-operator/upstream | [v1.8.0-rc.1](https://github.com/kubeflow/training-operator/tree/v1.8.0-rc.1/manifests) | -| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/notebook-controller/config) | -| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/pvcviewer-controller/config) | -| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/tensorboard-controller/config) | -| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/centraldashboard/manifests) | -| Profiles + KFAM | apps/profiles/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/profile-controller/config) | -| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/admission-webhook/manifests) | -| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/crud-web-apps/jupyter/manifests) | -| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/crud-web-apps/tensorboards/manifests) | -| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.1](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.1/components/crud-web-apps/volumes/manifests) | +| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/notebook-controller/config) | +| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/pvcviewer-controller/config) | +| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/tensorboard-controller/config) | +| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/centraldashboard/manifests) | +| Profiles + KFAM | apps/profiles/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/profile-controller/config) | +| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/admission-webhook/manifests) | +| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/jupyter/manifests) | +| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/tensorboards/manifests) | +| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/volumes/manifests) | | Katib | apps/katib/upstream | [v0.17.0-rc.0](https://github.com/kubeflow/katib/tree/v0.17.0-rc.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | | KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | diff --git a/apps/admission-webhook/upstream/base/kustomization.yaml b/apps/admission-webhook/upstream/base/kustomization.yaml index 6106121953..2b70cdaeb4 100644 --- a/apps/admission-webhook/upstream/base/kustomization.yaml +++ b/apps/admission-webhook/upstream/base/kustomization.yaml @@ -16,7 +16,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/poddefaults-webhook newName: docker.io/kubeflownotebookswg/poddefaults-webhook - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 namespace: kubeflow generatorOptions: disableNameSuffixHash: true diff --git a/apps/centraldashboard/upstream/base/kustomization.yaml b/apps/centraldashboard/upstream/base/kustomization.yaml index d7164b27b9..99cdab21b1 100644 --- a/apps/centraldashboard/upstream/base/kustomization.yaml +++ b/apps/centraldashboard/upstream/base/kustomization.yaml @@ -13,7 +13,7 @@ resources: images: - name: docker.io/kubeflownotebookswg/centraldashboard newName: docker.io/kubeflownotebookswg/centraldashboard - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 configMapGenerator: - envs: - params.env diff --git a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml index 9082504839..c03654976b 100644 --- a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml +++ b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml @@ -23,7 +23,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/jupyter-web-app newName: docker.io/kubeflownotebookswg/jupyter-web-app - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml index 03be6e6f00..e1d4830ea2 100644 --- a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml +++ b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml @@ -5,4 +5,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/notebook-controller newName: docker.io/kubeflownotebookswg/notebook-controller - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 diff --git a/apps/profiles/upstream/base/kustomization.yaml b/apps/profiles/upstream/base/kustomization.yaml index b8cafde069..0d29098f03 100644 --- a/apps/profiles/upstream/base/kustomization.yaml +++ b/apps/profiles/upstream/base/kustomization.yaml @@ -12,7 +12,7 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/profile-controller newName: docker.io/kubeflownotebookswg/profile-controller - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 configMapGenerator: - name: namespace-labels-data diff --git a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml index 940bc7e237..e68ee70492 100644 --- a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml +++ b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml @@ -29,4 +29,4 @@ vars: images: - name: docker.io/kubeflownotebookswg/kfam newName: docker.io/kubeflownotebookswg/kfam - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 diff --git a/apps/pvcviewer-controller/upstream/base/kustomization.yaml b/apps/pvcviewer-controller/upstream/base/kustomization.yaml index 7a3af06ebf..4eb0ae99ac 100644 --- a/apps/pvcviewer-controller/upstream/base/kustomization.yaml +++ b/apps/pvcviewer-controller/upstream/base/kustomization.yaml @@ -6,4 +6,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/pvcviewer-controller newName: docker.io/kubeflownotebookswg/pvcviewer-controller - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 diff --git a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml index 275c4cebc3..c16af71f59 100644 --- a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml @@ -14,4 +14,4 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/tensorboard-controller newName: docker.io/kubeflownotebookswg/tensorboard-controller - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 diff --git a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml index dc99f3049c..3d6596f1f3 100644 --- a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/tensorboards-web-app newName: docker.io/kubeflownotebookswg/tensorboards-web-app - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/volumes-web-app/upstream/base/kustomization.yaml b/apps/volumes-web-app/upstream/base/kustomization.yaml index aaaf834ecc..c2343baa1c 100644 --- a/apps/volumes-web-app/upstream/base/kustomization.yaml +++ b/apps/volumes-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/volumes-web-app newName: docker.io/kubeflownotebookswg/volumes-web-app - newTag: v1.9.0-rc.1 + newTag: v1.9.0-rc.2 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/hack/synchronize-kubeflow-manifests.sh b/hack/synchronize-kubeflow-manifests.sh index edaff2e0b3..96c5154f47 100644 --- a/hack/synchronize-kubeflow-manifests.sh +++ b/hack/synchronize-kubeflow-manifests.sh @@ -14,7 +14,7 @@ set -euo pipefail IFS=$'\n\t' -COMMIT="v1.9.0-rc.1" # You can use tags as well +COMMIT="v1.9.0-rc.2" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-kubeflow} BRANCH=${BRANCH:=synchronize-kubeflow-kubeflow-manifests-${COMMIT?}} From da2b718cb1335a7eb3da93be2d566878b4119f14 Mon Sep 17 00:00:00 2001 From: Diego Lovison Date: Tue, 9 Jul 2024 11:40:12 -0300 Subject: [PATCH 06/16] Remove --kubeconfig mycluster.yaml and use KUBECONFIG instead of moving files (#2792) Signed-off-by: Diego Lovison --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 99d3e9a3e0..e223f489ee 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,7 @@ The `example` directory contains an example kustomization for the single command #### Create kind cluster ```sh -cat < ~/.kube/config +kind get kubeconfig --name kubeflow > /tmp/kubeflow-config +export KUBECONFIG=/tmp/kubeflow-config ``` #### Create a Secret based on existing credentials in order to pull the images From 9b22cdce54389fcd064bd989dc8eae797093b6a0 Mon Sep 17 00:00:00 2001 From: Hansini Karunarathne <107214435+hansinikarunarathne@users.noreply.github.com> Date: Tue, 9 Jul 2024 20:22:12 +0530 Subject: [PATCH 07/16] Introduce proper formatting on python ,bash and yaml files (#2774) * Add balck github action to format python files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Formatted python files from the balck formatter Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add github action to format yaml files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add step YAML Formatting Guidelines to yaml_formatter.yaml file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made chnages to run next steps although the previous step fail Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * switch steps Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * added shellCheck for bash formatting Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Changed code only to lint yaml files inside the common and example folder Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Changed main to master Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Run yamllint on files which is chnaged in PR only Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Changed 'Proper formatting on python files' github workflow only to run for python files in common and example folder Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add shellcheckrc file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Disable SC1017,SC2086,SC2070 rules when shellcheck and did changes in sh files to address the formattings according to shellcheck Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made chnages in sh files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made chnages in sh files to address formattings Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made chnages in sh files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * disable SC2046 rule Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * disable SC2006 and SC2155 rule Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Change shellcheck only to run for PR chnaged files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Address a issue with bash_formatter.yaml Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Changed origin/main to origin/master Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Formatted python files from the balck formatter Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add github action to format yaml files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add step YAML Formatting Guidelines to yaml_formatter.yaml file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made chnages to run next steps although the previous step fail Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * switch steps Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add shellcheckrc file Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Disable SC1017,SC2086,SC2070 rules when shellcheck and did changes in sh files to address the formattings according to shellcheck Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * made chnages in sh files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Change shellcheck only to run for PR chnaged files Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Ensure the full history is fetched Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Ensure the full history is fetched in bash_formatter Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * fix SC2148 (error) Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * remove commented lines in bash_formatter.yaml Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * remove formatting chnages done to app folder content Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add guidlines to how to format python files according to black formatter Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> * Add guidlines to how to format bash files according to shellcheck formatter Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --------- Signed-off-by: hansinikarunarathne <107214435+hansinikarunarathne@users.noreply.github.com> --- .github/workflows/bash_formatter.yaml | 40 ++++ .github/workflows/python_formatter.yaml | 32 +++ .github/workflows/yaml_formatter.yaml | 59 +++++ .shellcheckrc | 2 + .yamllint.yaml | 12 + contrib/kserve/tests/test_sklearn.py | 4 +- hack/synchronize-istio-cni-manifests.sh | 2 +- hack/synchronize-istio-manifests.sh | 3 +- hack/synchronize-kserve-web-app-manifests.sh | 8 +- hack/trivy_scan.py | 220 ++++++++++++------- run_yamllint.sh | 10 + tests/gh-actions/kf-objects/test_pipeline.py | 17 +- tests/gh-actions/runasnonroot.sh | 204 ++++++++--------- 13 files changed, 422 insertions(+), 191 deletions(-) create mode 100644 .github/workflows/bash_formatter.yaml create mode 100644 .github/workflows/python_formatter.yaml create mode 100644 .github/workflows/yaml_formatter.yaml create mode 100644 .shellcheckrc create mode 100644 .yamllint.yaml create mode 100644 run_yamllint.sh diff --git a/.github/workflows/bash_formatter.yaml b/.github/workflows/bash_formatter.yaml new file mode 100644 index 0000000000..fd1d254bf6 --- /dev/null +++ b/.github/workflows/bash_formatter.yaml @@ -0,0 +1,40 @@ +name: Proper Formatting on bash files + +on: [push, pull_request] + +jobs: + format_bash_files: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: Install ShellCheck + run: sudo apt install -y shellcheck + + - name: Bash Formatting Guidelines + run: | + echo "### Bash Files Formatting Guidelines ### + If there are errors and warnings regarding your bash files, + You can check that error code definitions in https://www.shellcheck.net/wiki/ site. + You can correct them using the https://www.shellcheck.net/ site. + You have to ignore disable errors in .shellcheckrc file. + " + + - name: Fetch master branch + run: git fetch origin master + + - name: Set up changed files + run: | + git diff --name-only origin/master...HEAD | grep -E '^.*\.sh$' | grep -v '^apps/' > changed_files_in_PR.txt || true + + - name: Display changed files + run: cat changed_files_in_PR.txt + + - name: Run ShellCheck on changed files + run: | + cat changed_files_in_PR.txt | xargs -I {} shellcheck {} + shell: bash + + diff --git a/.github/workflows/python_formatter.yaml b/.github/workflows/python_formatter.yaml new file mode 100644 index 0000000000..ba470cad84 --- /dev/null +++ b/.github/workflows/python_formatter.yaml @@ -0,0 +1,32 @@ +name: Proper Formatting on Python files + +on: [push, pull_request] + +jobs: + format_python_files: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Python Files Formatting Guidelines + run: | + echo "### Python Files Formatting Guidelines ### + If there is a formatting errors in your python files, + 1. First install black + It requires Python 3.8+ to run. + Install with "pip install black" and if you use pipx, install Black with "pipx install black" + If you want to format Jupyter Notebooks, install with pip install "black[jupyter]" + + 2. Run the command + "python -m black {source_file_or_directory}" or + "black {source_file_or_directory}" + to format python files. + " + + - uses: psf/black@stable + with: + src: | + ./common + ./example + + \ No newline at end of file diff --git a/.github/workflows/yaml_formatter.yaml b/.github/workflows/yaml_formatter.yaml new file mode 100644 index 0000000000..dca73409d4 --- /dev/null +++ b/.github/workflows/yaml_formatter.yaml @@ -0,0 +1,59 @@ +name: Proper Formatting on YAML files + +on: [push, pull_request] + +jobs: + format_YAML_files: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + + - name: Install yamllint + run: pip install yamllint + + - name: YAML Formatting Guidelines + run: | + echo "### YAML Formatting Guidelines ### + If there is a formatting error in your YAML file, you will see errors like the one below: + 'Error: 6:4 [indentation] wrong indentation: expected 2 but found 3' + + To fix these errors, refer to the YAML formatting rules at: + https://yamllint.readthedocs.io/en/stable/rules.html# + + Search for the keyword inside the brackets [] in the error message. In this example, it's 'indentation'. + + Note: Some rules have been customized in the '.yamllint.yaml' file. Below is the content of that file: + + extends: default + + rules: + document-start: + present: false + document-end: + present: false + indentation: + indent-sequences: false + line-length: + max: 400 + " + + - name: Fetch master branch + run: git fetch origin master + + - name: Set up changed files + run: | + git diff --name-only origin/master...HEAD | grep -E '^common/.*\.ya?ml$|^example/.*\.ya?ml$' > changed_files_in_PR.txt || true + + - name: Display changed files + run: cat changed_files_in_PR.txt + + - name: Run yamllint on changed files + run: | + chmod +x ./run_yamllint.sh + ./run_yamllint.sh + shell: bash + + + diff --git a/.shellcheckrc b/.shellcheckrc new file mode 100644 index 0000000000..9c339b78ba --- /dev/null +++ b/.shellcheckrc @@ -0,0 +1,2 @@ +# ~/.shellcheckrc +disable=SC1017,SC2086,SC2070,SC2046,SC2155,SC2006 diff --git a/.yamllint.yaml b/.yamllint.yaml new file mode 100644 index 0000000000..46153d48f6 --- /dev/null +++ b/.yamllint.yaml @@ -0,0 +1,12 @@ +# It extends the default conf by adjusting some options. +extends: default + +rules: + document-start: + present: false + document-end: + present: false + indentation: + indent-sequences: false + line-length: + max: 400 \ No newline at end of file diff --git a/contrib/kserve/tests/test_sklearn.py b/contrib/kserve/tests/test_sklearn.py index 2c17257019..0e2d21241c 100644 --- a/contrib/kserve/tests/test_sklearn.py +++ b/contrib/kserve/tests/test_sklearn.py @@ -50,7 +50,9 @@ def test_sklearn_kserve(): spec=V1beta1InferenceServiceSpec(predictor=predictor), ) - kserve_client = KServeClient(config_file=os.environ.get("KUBECONFIG", "~/.kube/config")) + kserve_client = KServeClient( + config_file=os.environ.get("KUBECONFIG", "~/.kube/config") + ) kserve_client.create(isvc) kserve_client.wait_isvc_ready(service_name, namespace=KSERVE_TEST_NAMESPACE) res = predict(service_name, "./data/iris_input.json") diff --git a/hack/synchronize-istio-cni-manifests.sh b/hack/synchronize-istio-cni-manifests.sh index b2fcd7c3af..1f8a22c3e5 100644 --- a/hack/synchronize-istio-cni-manifests.sh +++ b/hack/synchronize-istio-cni-manifests.sh @@ -1,5 +1,5 @@ -@ -1,88 +0,0 @@ #!/usr/bin/env bash +@ -1,88 +0,0 @@ # This script aims at helping create a PR to update the manifests of the # knative. diff --git a/hack/synchronize-istio-manifests.sh b/hack/synchronize-istio-manifests.sh index 6a4c8987de..ad75d68022 100644 --- a/hack/synchronize-istio-manifests.sh +++ b/hack/synchronize-istio-manifests.sh @@ -1,4 +1,5 @@ -# #!/usr/bin/env bash +#!/usr/bin/env bash + # # This script aims at helping create a PR to update the manifests of Istio # # This script: diff --git a/hack/synchronize-kserve-web-app-manifests.sh b/hack/synchronize-kserve-web-app-manifests.sh index f7b5b102b2..beabded784 100644 --- a/hack/synchronize-kserve-web-app-manifests.sh +++ b/hack/synchronize-kserve-web-app-manifests.sh @@ -24,7 +24,7 @@ if [ -n "$(git status --porcelain)" ]; then echo "WARNING: You have uncommitted changes" fi -if [ `git branch --list $BRANCH` ] +if [ "$(git branch --list $BRANCH)" ] then echo "WARNING: Branch $BRANCH already exists." fi @@ -39,11 +39,11 @@ echo "Checking out in $SRC_DIR to $COMMIT..." # Checkout the Model Registry repository mkdir -p $SRC_DIR -cd $SRC_DIR +cd $SRC_DIR || exit if [ ! -d "models-web-app/.git" ]; then git clone https://github.com/kserve/models-web-app.git fi -cd $SRC_DIR/models-web-app +cd $SRC_DIR/models-web-app || exit if ! git rev-parse --verify --quiet $COMMIT; then git checkout -b $COMMIT else @@ -71,7 +71,7 @@ DST_TXT="\[$COMMIT\](https://github.com/kserve/models-web-app/tree/$COMMIT/confi sed -i "s|$SRC_TXT|$DST_TXT|g" "${MANIFESTS_DIR}"/README.md echo "Committing the changes..." -cd $MANIFESTS_DIR +cd $MANIFESTS_DIR || exit git add contrib/kserve/models-web-app git add README.md git commit -s -m "Update kserve models web application manifests from ${COMMIT}" diff --git a/hack/trivy_scan.py b/hack/trivy_scan.py index d76536604f..ff7706bb1b 100644 --- a/hack/trivy_scan.py +++ b/hack/trivy_scan.py @@ -7,8 +7,8 @@ # - Summary of security counts with images a JSON file inside ../image_lists/summary_of_severity_counts_for_WG folder # 4. Generate a summary of the security scan reports # - The summary will be saved in JSON format inside ../image_lists/summary_of_severity_counts_for_WG folder -# 5. Before run this file you have to -# 1. Install kustomize +# 5. Before run this file you have to +# 1. Install kustomize # - sudo apt install snapd # - sudo snap install kustomize # 2. Install trivy @@ -37,31 +37,36 @@ "manifests": "../common/cert-manager/cert-manager/base ../common/cert-manager/kubeflow-issuer/base ../common/istio-1-22/istio-crds/base ../common/istio-1-22/istio-namespace/base ../common/istio-1-22/istio-install/overlays/oauth2-proxy ../common/oidc-client/oauth2-proxy/overlays/m2m-self-signed ../common/dex/overlays/oauth2-proxy ../common/knative/knative-serving/overlays/gateways ../common/knative/knative-eventing/base ../common/istio-1-22/cluster-local-gateway/base ../common/kubeflow-namespace/base ../common/kubeflow-roles/base ../common/istio-1-22/kubeflow-istio-resources/base", "workbenches": "../apps/pvcviewer-controller/upstream/base ../apps/admission-webhook/upstream/overlays ../apps/centraldashboard/upstream/overlays/oauth2-proxy ../apps/jupyter/jupyter-web-app/upstream/overlays ../apps/volumes-web-app/upstream/overlays ../apps/tensorboard/tensorboards-web-app/upstream/overlays ../apps/profiles/upstream/overlays ../apps/jupyter/notebook-controller/upstream/overlays ../apps/tensorboard/tensorboard-controller/upstream/overlays", "serving": "../contrib/kserve - ../contrib/kserve/models-web-app/overlays/kubeflow", - "model-registry": "../apps/model-registry/upstream" + "model-registry": "../apps/model-registry/upstream", } DIRECTORY = "../image_lists" os.makedirs(DIRECTORY, exist_ok=True) SCAN_REPORTS_DIR = os.path.join(DIRECTORY, "security_scan_reports") ALL_SEVERITY_COUNTS = os.path.join(DIRECTORY, "severity_counts_with_images_for_WG") -SUMMARY_OF_SEVERITY_COUNTS = os.path.join(DIRECTORY, "summary_of_severity_counts_for_WG") +SUMMARY_OF_SEVERITY_COUNTS = os.path.join( + DIRECTORY, "summary_of_severity_counts_for_WG" +) os.makedirs(SCAN_REPORTS_DIR, exist_ok=True) os.makedirs(ALL_SEVERITY_COUNTS, exist_ok=True) os.makedirs(SUMMARY_OF_SEVERITY_COUNTS, exist_ok=True) + def log(*args, **kwargs): # Custom log function that print messages with flush=True by default. - kwargs.setdefault('flush', True) + kwargs.setdefault("flush", True) print(*args, **kwargs) + def save_images(wg, images, version): # Saves a list of container images to a text file named after the workgroup and version. output_file = f"../image_lists/kf_{version}_{wg}_images.txt" - with open(output_file, 'w') as f: - f.write('\n'.join(images)) + with open(output_file, "w") as f: + f.write("\n".join(images)) log(f"File {output_file} successfully created") + def validate_semantic_version(version): # Validates a semantic version string (e.g., "0.1.2" or "latest"). regex = r"^[0-9]+\.[0-9]+\.[0-9]+$" @@ -70,29 +75,46 @@ def validate_semantic_version(version): else: raise ValueError(f"Invalid semantic version: '{version}'") + def extract_images(version): version = validate_semantic_version(version) log(f"Running the script using Kubeflow version: {version}") - all_images = set() # Collect all unique images across workgroups + all_images = set() # Collect all unique images across workgroups for wg, dirs in wg_dirs.items(): wg_images = set() # Collect unique images for this workgroup for dir_path in dirs.split(): for root, _, files in os.walk(dir_path): for file in files: - if file in ["kustomization.yaml", "kustomization.yml", "Kustomization"]: + if file in [ + "kustomization.yaml", + "kustomization.yml", + "Kustomization", + ]: full_path = os.path.join(root, file) try: # Execute `kustomize build` to render the kustomization file - result = subprocess.run(['kustomize', 'build', root], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) + result = subprocess.run( + ["kustomize", "build", root], + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) except subprocess.CalledProcessError as e: - log(f"ERROR:\t Failed \"kustomize build\" command for directory: {root}. See error above") + log( + f'ERROR:\t Failed "kustomize build" command for directory: {root}. See error above' + ) continue - + # Use regex to find lines with 'image: :' or 'image: ' # and '- image: :' but avoid environment variables - kustomize_images = re.findall(r'^\s*-?\s*image:\s*([^$\s:]+(?:\:[^\s]+)?)$', result.stdout, re.MULTILINE) + kustomize_images = re.findall( + r"^\s*-?\s*image:\s*([^$\s:]+(?:\:[^\s]+)?)$", + result.stdout, + re.MULTILINE, + ) wg_images.update(kustomize_images) # Ensure uniqueness within workgroup images @@ -104,24 +126,36 @@ def extract_images(version): uniq_images = sorted(all_images) save_images("all", uniq_images, version) -parser = argparse.ArgumentParser(description="Extract images from Kubeflow kustomizations.") + +parser = argparse.ArgumentParser( + description="Extract images from Kubeflow kustomizations." +) # Define a positional argument 'version' with optional occurrence and default value 'latest'. You can run this file as python3 .py or python .py -parser.add_argument("version", nargs="?", type=str, default="latest", help="Kubeflow version to use (defaults to latest).") +parser.add_argument( + "version", + nargs="?", + type=str, + default="latest", + help="Kubeflow version to use (defaults to latest).", +) args = parser.parse_args() extract_images(args.version) - log("Started scanning images") # Get list of text files excluding "kf_latest_all_images.txt" -files = [f for f in glob.glob(os.path.join(DIRECTORY, "*.txt")) if not f.endswith("kf_latest_all_images.txt")] +files = [ + f + for f in glob.glob(os.path.join(DIRECTORY, "*.txt")) + if not f.endswith("kf_latest_all_images.txt") +] # Loop through each text file in the specified directory for file in files: log(f"Scanning images in {file}") - file_base_name = os.path.basename(file).replace('.txt', '') + file_base_name = os.path.basename(file).replace(".txt", "") # Directory to save reports for this specific file file_reports_dir = os.path.join(SCAN_REPORTS_DIR, file_base_name) @@ -131,68 +165,86 @@ def extract_images(version): severity_count = os.path.join(file_reports_dir, "severity_counts") os.makedirs(severity_count, exist_ok=True) - with open(file, 'r') as f: + with open(file, "r") as f: lines = f.readlines() for line in lines: line = line.strip() - image_name = line.split(':')[0] - image_tag = line.split(':')[1] if ':' in line else '' + image_name = line.split(":")[0] + image_tag = line.split(":")[1] if ":" in line else "" - image_name_scan = image_name.split('/')[-1] + image_name_scan = image_name.split("/")[-1] if image_tag: image_name_scan = f"{image_name_scan}_{image_tag}" - - scan_output_file = os.path.join(file_reports_dir, f"{image_name_scan}_scan.json") - log(f"Scanning ",line) + scan_output_file = os.path.join( + file_reports_dir, f"{image_name_scan}_scan.json" + ) - try: - result = subprocess.run(["trivy", "image", "--format", "json", "--output", scan_output_file, line], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) + log(f"Scanning ", line) - with open(scan_output_file, 'r') as json_file: + try: + result = subprocess.run( + [ + "trivy", + "image", + "--format", + "json", + "--output", + scan_output_file, + line, + ], + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + with open(scan_output_file, "r") as json_file: scan_data = json.load(json_file) - if not scan_data.get('Results'): + if not scan_data.get("Results"): log(f"No vulnerabilities found in {image_name}:{image_tag}") else: vulnerabilities_list = [ - result['Vulnerabilities'] - for result in scan_data['Results'] - if 'Vulnerabilities' in result and result['Vulnerabilities'] + result["Vulnerabilities"] + for result in scan_data["Results"] + if "Vulnerabilities" in result and result["Vulnerabilities"] ] if not vulnerabilities_list: - log(f"The vulnerabilities detection may be insufficient because security updates are not provided for {image_name}:{image_tag}\n") + log( + f"The vulnerabilities detection may be insufficient because security updates are not provided for {image_name}:{image_tag}\n" + ) else: severity_counts = {"LOW": 0, "MEDIUM": 0, "HIGH": 0, "CRITICAL": 0} for vulnerabilities in vulnerabilities_list: for vulnerability in vulnerabilities: - severity = vulnerability.get('Severity', 'UNKNOWN') - if severity == 'UNKNOWN': + severity = vulnerability.get("Severity", "UNKNOWN") + if severity == "UNKNOWN": continue elif severity in severity_counts: severity_counts[severity] += 1 - - report = { - "image": line, - "severity_counts": severity_counts - } + report = {"image": line, "severity_counts": severity_counts} image_table = PrettyTable() image_table.field_names = ["Critical", "High", "Medium", "Low"] - image_table.add_row([ + image_table.add_row( + [ severity_counts["CRITICAL"], severity_counts["HIGH"], severity_counts["MEDIUM"], - severity_counts["LOW"] - ]) + severity_counts["LOW"], + ] + ) log(f"{image_table}\n") - severity_report_file = os.path.join(severity_count, f"{image_name_scan}_severity_report.json") - with open(severity_report_file, 'w') as report_file: + severity_report_file = os.path.join( + severity_count, f"{image_name_scan}_severity_report.json" + ) + with open(severity_report_file, "w") as report_file: json.dump(report, report_file, indent=4) except subprocess.CalledProcessError as e: @@ -209,16 +261,18 @@ def extract_images(version): else: combined_data = [] for json_file in json_files: - with open(json_file, 'r') as jf: + with open(json_file, "r") as jf: combined_data.append(json.load(jf)) - with open(output_file, 'w') as of: + with open(output_file, "w") as of: json.dump({"data": combined_data}, of, indent=4) log(f"JSON files successfully combined into '{output_file}'") # File to save summary of the severity counts for WGs as JSON format. -summary_file = os.path.join(SUMMARY_OF_SEVERITY_COUNTS, "severity_summary_in_json_format.json") +summary_file = os.path.join( + SUMMARY_OF_SEVERITY_COUNTS, "severity_summary_in_json_format.json" +) # Initialize counters total_images = 0 @@ -233,27 +287,27 @@ def extract_images(version): # Loop through each JSON file in the ALL_SEVERITY_COUNTS for file_path in glob.glob(os.path.join(ALL_SEVERITY_COUNTS, "*.json")): # Split filename based on underscores - filename_parts = os.path.basename(file_path).split('_') + filename_parts = os.path.basename(file_path).split("_") # Check if there are at least 3 parts (prefix, name, _images) if len(filename_parts) >= 4: - # Extract name (second part) - filename = filename_parts[2] - filename = filename.capitalize() + # Extract name (second part) + filename = filename_parts[2] + filename = filename.capitalize() else: - log(f"Skipping invalid filename format: {file_path}") - continue + log(f"Skipping invalid filename format: {file_path}") + continue - with open(file_path, 'r') as f: - data = json.load(f)['data'] + with open(file_path, "r") as f: + data = json.load(f)["data"] # Initialize counts for this file image_count = len(data) - low = sum(entry['severity_counts']['LOW'] for entry in data) - medium = sum(entry['severity_counts']['MEDIUM'] for entry in data) - high = sum(entry['severity_counts']['HIGH'] for entry in data) - critical = sum(entry['severity_counts']['CRITICAL'] for entry in data) + low = sum(entry["severity_counts"]["LOW"] for entry in data) + medium = sum(entry["severity_counts"]["MEDIUM"] for entry in data) + high = sum(entry["severity_counts"]["HIGH"] for entry in data) + critical = sum(entry["severity_counts"]["CRITICAL"] for entry in data) # Update the total counts total_images += image_count @@ -268,67 +322,85 @@ def extract_images(version): "LOW": low, "MEDIUM": medium, "HIGH": high, - "CRITICAL": critical + "CRITICAL": critical, } # Update merged_data with filename as key merged_data[filename] = file_data # Add total counts to merged_data - merged_data['total'] = { + merged_data["total"] = { "images": total_images, "LOW": total_low, "MEDIUM": total_medium, "HIGH": total_high, - "CRITICAL": total_critical + "CRITICAL": total_critical, } log("Summary in Json Format:") -log(json.dumps(merged_data, indent=4)) +log(json.dumps(merged_data, indent=4)) # Write the final output to a file -with open(summary_file, 'w') as summary_f: +with open(summary_file, "w") as summary_f: json.dump(merged_data, summary_f, indent=4) log(f"Summary written to: {summary_file} as JSON format") # Load JSON content from the file -with open(summary_file, 'r') as file: +with open(summary_file, "r") as file: data = json.load(file) # Define a mapping for working group names groupnames = { "Automl": "AutoML", "Pipelines": "Pipelines", - "Workbenches":"Workbenches(Notebooks)", + "Workbenches": "Workbenches(Notebooks)", "Serving": "Kserve", - "Manifests":"Manifests", + "Manifests": "Manifests", "Training": "Training", - "Model-registry":"Model Registry", + "Model-registry": "Model Registry", "total": "All Images", } # Create PrettyTable table = PrettyTable() -table.field_names = ["Working Group", "Images", "Critical CVE", "High CVE", "Medium CVE", "Low CVE"] +table.field_names = [ + "Working Group", + "Images", + "Critical CVE", + "High CVE", + "Medium CVE", + "Low CVE", +] # Populate the table with data for group_name in groupnames: if group_name in data: # Check if group_name exists in data value = data[group_name] - table.add_row([groupnames[group_name], value["images"], value["CRITICAL"], value["HIGH"], value["MEDIUM"], value["LOW"]]) + table.add_row( + [ + groupnames[group_name], + value["images"], + value["CRITICAL"], + value["HIGH"], + value["MEDIUM"], + value["LOW"], + ] + ) # log the table log(table) # Write the table output to a file in the specified folder -output_file = SUMMARY_OF_SEVERITY_COUNTS + '/summary_of_severity_counts_for_WGs_in_table.txt' -with open(output_file, 'w') as f: +output_file = ( + SUMMARY_OF_SEVERITY_COUNTS + "/summary_of_severity_counts_for_WGs_in_table.txt" +) +with open(output_file, "w") as f: f.write(str(table)) log("Output saved to:", output_file) log("Severity counts with images respect to WGs are saved in the",ALL_SEVERITY_COUNTS) -log("Scanned Json reports on images are saved in" ,SCAN_REPORTS_DIR) \ No newline at end of file +log("Scanned Json reports on images are saved in",SCAN_REPORTS_DIR) \ No newline at end of file diff --git a/run_yamllint.sh b/run_yamllint.sh new file mode 100644 index 0000000000..3225710f59 --- /dev/null +++ b/run_yamllint.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +if [ -s changed_files_in_PR.txt ]; then + while IFS= read -r file; do + echo "Running yamllint on $file" + yamllint "$file" + done < changed_files_in_PR.txt +else + echo "No YAML files changed in this PR." +fi diff --git a/tests/gh-actions/kf-objects/test_pipeline.py b/tests/gh-actions/kf-objects/test_pipeline.py index 9bd8228e5a..6755d30ff4 100755 --- a/tests/gh-actions/kf-objects/test_pipeline.py +++ b/tests/gh-actions/kf-objects/test_pipeline.py @@ -7,22 +7,23 @@ def echo_op(): print("Test pipeline") -@dsl.pipeline( - name='test-pipeline', - description='A test pipeline.' -) + +@dsl.pipeline(name="test-pipeline", description="A test pipeline.") def hello_world_pipeline(): echo_task = echo_op() + if __name__ == "__main__": # Run the Kubeflow Pipeline in the user's namespace. - kfp_client = kfp.Client(host="http://localhost:3000", - namespace="kubeflow-user-example-com") + kfp_client = kfp.Client( + host="http://localhost:3000", namespace="kubeflow-user-example-com" + ) kfp_client.runs.api_client.default_headers.update( - {"kubeflow-userid": "kubeflow-user-example-com"}) + {"kubeflow-userid": "kubeflow-user-example-com"} + ) # create the KFP run run_id = kfp_client.create_run_from_pipeline_func( hello_world_pipeline, namespace="kubeflow-user-example-com", arguments={}, - ).run_id \ No newline at end of file + ).run_id diff --git a/tests/gh-actions/runasnonroot.sh b/tests/gh-actions/runasnonroot.sh index 921623c9c8..532424cd37 100644 --- a/tests/gh-actions/runasnonroot.sh +++ b/tests/gh-actions/runasnonroot.sh @@ -1,102 +1,102 @@ -#!/bin/bash - -namespace="kubeflow" -error_flag=0 - -# Function to check if 'id' command is available in a container -has_id_command() { - local pod_name="$1" - local container_name="$2" - - # Execute 'id' command and capture the output - if kubectl exec -it -n "$namespace" "$pod_name" -c "$container_name" -- id -u >/dev/null 2>&1; then - return 0 # 'id' command is available - else - return 1 # 'id' command is not available - fi -} - -# Function to check 'securityContext' and 'runAsNonRoot' at the pod or container level -has_securityContext_and_runAsNonRoot() { - local pod_name="$1" - local container_name="$2" - - # Use jq to check if 'securityContext' is defined at the pod level - local securityContextPod=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.securityContext') - - if [ "$securityContextPod" = "null" ]; then - : # 'securityContext' is missing at the pod level, continue checking at the container level - else - # Check 'runAsNonRoot' at the pod level - local runAsNonRootPod=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.securityContext.runAsNonRoot // "Missing"') - - if [ "$runAsNonRootPod" = "Missing" ]; then - : # 'runAsNonRoot' is missing at the pod level, continue checking at the container level - else - return 0 # 'runAsNonRoot' is present at the pod level (success) - fi - fi - - # Use jq to check 'securityContext' at the container level - local securityContextContainer=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.containers[] | select(.name == "'"$container_name"'").securityContext') - - if [ "$securityContextContainer" = "null" ]; then - if [ "$securityContextPod" = "null" ]; then - echo "Error: 'securityContext' is missing at the pod and container level in container $container_name of pod $pod_name" - return 1 - else - echo "Error: There is no runasnonroot on pod level and 'securityContext' is missing at container level in container $container_name of pod $pod_name" - return 1 - fi - fi - - # Check 'runAsNonRoot' at the container level - local runAsNonRootContainer=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.containers[] | select(.name == "'"$container_name"'").securityContext.runAsNonRoot // "Missing"') - - if [ "$runAsNonRootContainer" = "Missing" ]; then - echo "Error: There is no runasnonroot on pod level and'runAsNonRoot' is missing in container $container_name of pod $pod_name" - return 1 # 'runAsNonRoot' is missing at the container level (fail) - fi - - return 0 # 'securityContext' and 'runAsNonRoot' are defined at the container level -} - -# Get a list of pod names in the specified namespace that are not in the "Completed" state -pod_names=$(kubectl get pods -n "$namespace" --field-selector=status.phase!=Succeeded,status.phase!=Failed -o json | jq -r '.items[].metadata.name') - -# Loop through the pod names and execute checks -for pod_name in $pod_names; do - echo "Entering pod $pod_name in namespace $namespace..." - - container_names=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.containers[].name') - - for container_name in $container_names; do - if has_securityContext_and_runAsNonRoot "$pod_name" "$container_name"; then - error_flag=1 - fi - - if has_id_command "$pod_name" "$container_name"; then - user_id=$(kubectl exec -it -n "$namespace" "$pod_name" -c "$container_name" -- id -u) - - # Clean up whitespace in the user_id using tr - user_id_cleaned=$(echo -n "$user_id" | tr -d '[:space:]') - - if [ "$user_id_cleaned" = "0" ]; then - echo "Error: Pod $pod_name contains user ID 0 in container $container_name" - error_flag=1 - else - echo "Container: $container_name - User ID: $user_id_cleaned" - fi - else - echo "Warning: 'id' command not available in container $container_name" - fi - done -done - -# Exit with an error if any pod contains an error condition -if [ $error_flag -eq 1 ]; then - exit 1 -fi - -# Exit successfully -exit 0 +#!/bin/bash + +namespace="kubeflow" +error_flag=0 + +# Function to check if 'id' command is available in a container +has_id_command() { + local pod_name="$1" + local container_name="$2" + + # Execute 'id' command and capture the output + if kubectl exec -it -n "$namespace" "$pod_name" -c "$container_name" -- id -u >/dev/null 2>&1; then + return 0 # 'id' command is available + else + return 1 # 'id' command is not available + fi +} + +# Function to check 'securityContext' and 'runAsNonRoot' at the pod or container level +has_securityContext_and_runAsNonRoot() { + local pod_name="$1" + local container_name="$2" + + # Use jq to check if 'securityContext' is defined at the pod level + local securityContextPod=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.securityContext') + + if [ "$securityContextPod" = "null" ]; then + : # 'securityContext' is missing at the pod level, continue checking at the container level + else + # Check 'runAsNonRoot' at the pod level + local runAsNonRootPod=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.securityContext.runAsNonRoot // "Missing"') + + if [ "$runAsNonRootPod" = "Missing" ]; then + : # 'runAsNonRoot' is missing at the pod level, continue checking at the container level + else + return 0 # 'runAsNonRoot' is present at the pod level (success) + fi + fi + + # Use jq to check 'securityContext' at the container level + local securityContextContainer=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.containers[] | select(.name == "'"$container_name"'").securityContext') + + if [ "$securityContextContainer" = "null" ]; then + if [ "$securityContextPod" = "null" ]; then + echo "Error: 'securityContext' is missing at the pod and container level in container $container_name of pod $pod_name" + return 1 + else + echo "Error: There is no runasnonroot on pod level and 'securityContext' is missing at container level in container $container_name of pod $pod_name" + return 1 + fi + fi + + # Check 'runAsNonRoot' at the container level + local runAsNonRootContainer=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.containers[] | select(.name == "'"$container_name"'").securityContext.runAsNonRoot // "Missing"') + + if [ "$runAsNonRootContainer" = "Missing" ]; then + echo "Error: There is no runasnonroot on pod level and'runAsNonRoot' is missing in container $container_name of pod $pod_name" + return 1 # 'runAsNonRoot' is missing at the container level (fail) + fi + + return 0 # 'securityContext' and 'runAsNonRoot' are defined at the container level +} + +# Get a list of pod names in the specified namespace that are not in the "Completed" state +pod_names=$(kubectl get pods -n "$namespace" --field-selector=status.phase!=Succeeded,status.phase!=Failed -o json | jq -r '.items[].metadata.name') + +# Loop through the pod names and execute checks +for pod_name in $pod_names; do + echo "Entering pod $pod_name in namespace $namespace..." + + container_names=$(kubectl get pod -n "$namespace" "$pod_name" -o json | jq -r '.spec.containers[].name') + + for container_name in $container_names; do + if has_securityContext_and_runAsNonRoot "$pod_name" "$container_name"; then + error_flag=1 + fi + + if has_id_command "$pod_name" "$container_name"; then + user_id=$(kubectl exec -it -n "$namespace" "$pod_name" -c "$container_name" -- id -u) + + # Clean up whitespace in the user_id using tr + user_id_cleaned=$(echo -n "$user_id" | tr -d '[:space:]') + + if [ "$user_id_cleaned" = "0" ]; then + echo "Error: Pod $pod_name contains user ID 0 in container $container_name" + error_flag=1 + else + echo "Container: $container_name - User ID: $user_id_cleaned" + fi + else + echo "Warning: 'id' command not available in container $container_name" + fi + done +done + +# Exit with an error if any pod contains an error condition +if [ $error_flag -eq 1 ]; then + exit 1 +fi + +# Exit successfully +exit 0 From 8e4fafba65a7cdf56528f7f720716651ca72d675 Mon Sep 17 00:00:00 2001 From: Diego Lovison Date: Wed, 10 Jul 2024 07:48:13 -0300 Subject: [PATCH 08/16] Simplify the pull request template (#2793) Simplify the pull request template Signed-off-by: Diego Lovison --- .github/PULL_REQUEST_TEMPLATE.md | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index d03b3e8457..2de7cae887 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,36 +1,20 @@ # Pull Request Template for Kubeflow manifests Issues -- Please include a summary of changes and the related issue. -- List any dependencies that are required for this change. -- Please delete the options that are not relevant. -- The following checklist will help you to satisfy the requirements. - - - ## ✏️ A brief description of the changes > I changed ... ## 📦 List any dependencies that are required for this change > My PR depends on # -## 🐛 If this PR is related to an issue, please put the link of the issue here. +## 🐛 If this PR is related to an issue, please put the link to the issue here. > The following issues are related, because ... - - -## ✅ Unit Test Checklist - - - [ ] 🛠️ Make sure you have installed kustomize == 5.2.1+ - - [ ] ✍️ Have you written new tests for your core changes, as applicable? - - [ ] 🔄 Have you successfully run existing tests with your changes ? - - [ ] 🚀 Have you successfully run existing and new tests with your changes ? - ## ✅ Contributor checklist - - [ ] All the commits have been _signed-off_ (To pass the `DCO` check) - - [ ] Submit the [Contributor License Agreements](https://cla.developers.google.com/clas) (To pass the `cla/google` check) - + - Make sure you have tested with kustomize. See [Installation Prerequisites](https://github.com/kubeflow/manifests#prerequisites) + - All the commits have been _signed-off_ (To pass the `DCO` check) + - Submit the [Contributor License Agreements](https://cla.developers.google.com/clas) (To pass the `cla/google` check) --- > You can join the CNCF Slack and access our meetings at the [Kubeflow Community](https://www.kubeflow.org/docs/about/community/) website. Our channel on the CNCF Slack is here [**#kubeflow-platform**](https://app.slack.com/client/T08PSQ7BQ/C073W572LA2). - \ No newline at end of file + From 36c35d134c225ac376d2238ff2985c567be05afe Mon Sep 17 00:00:00 2001 From: StefanoFioravanzo Date: Wed, 10 Jul 2024 13:26:13 +0200 Subject: [PATCH 09/16] Document the CVE scanning process (#2785) * Document the CVE scanning process Signed-off-by: Stefano Fioravanzo * Add steps to find CVE logs Signed-off-by: Stefano Fioravanzo * Delete outdated documentation and add CVE scanning documentation. Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * Delete outdated documentation and add CVE scanning documentation. Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * Delete outdated documentation and add CVE scanning documentation. Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * Delete outdated documentation and add CVE scanning documentation. Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: Stefano Fioravanzo Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Co-authored-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 15 ++ documents/KustomizeBestPractices.md | 265 ---------------------------- 2 files changed, 15 insertions(+), 265 deletions(-) delete mode 100644 documents/KustomizeBestPractices.md diff --git a/README.md b/README.md index e223f489ee..3d0e4ca2c6 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ * [Change default user password](#change-default-user-password) - [Upgrading and extending](#upgrading-and-extending) - [Release process](#release-process) +- [CVE Scanning](#cve-scanning) - [Frequently Asked Questions](#frequently-asked-questions) @@ -502,6 +503,20 @@ The Manifest Working Group releases Kubeflow based on the [release timeline](htt and follow the [release versioning policy](https://github.com/kubeflow/community/blob/master/releases/handbook.md#versioning-policy), as defined in the [Kubeflow release handbook](https://github.com/kubeflow/community/blob/master/releases/handbook.md). +## CVE Scanning + +To view all past security scans, head to the [Image Extracting and Security Scanning GitHub Action workflow](https://github.com/kubeflow/manifests/actions/workflows/trivy.yaml). In the logs of the workflow you can expand the `Run image extracting and security scanning script` step to view the CVE logs. You will find a per-image CVE scan and a JSON dump of per-WorkingGroup aggregated metrics. +You can run the Python script from the workflow file locally on your machine to obtain the detailed JSON files for any git commit. + +The Kubeflow security working group follows a responsible disclosure policy for CVE results: + +- **Internal Review**: All CVE findings are initially reviewed internally by the security working group. +- **Severity Assessment**: Each CVE is assessed for severity and potential impact on the Kubeflow project. +- **Disclosure**: For high and critical severity CVEs, the security working group will: + - Notify the maintainers and contributors + - Try to provide a fix or mitigation strategy + - Publicly disclose the CVE details + ## Frequently Asked Questions - **Q:** What versions of Istio, Knative, Cert-Manager, Argo, ... are compatible with Kubeflow? \ diff --git a/documents/KustomizeBestPractices.md b/documents/KustomizeBestPractices.md deleted file mode 100644 index 9ee3651d6f..0000000000 --- a/documents/KustomizeBestPractices.md +++ /dev/null @@ -1,265 +0,0 @@ - -# Kustomize Best Practices - - This doc provides best practices for writing Kubeflow kustomize packages. - - - -**Table of Contents** - -- [Layout package to support composability](#layout-package-to-support-composability) -- [Reuse patches](#reuse-patches) - - [Disable security check for file outside of directory root](#disable-security-check-for-file-outside-of-directory-root) -- [Command Line substitution](#command-line-substitution) -- [Eschew vars](#eschew-vars) - - [Internal subsitution of fields Kustomize isn't aware of](#internal-subsitution-of-fields-kustomize-isnt-aware-of) - - [Global substitution](#global-substitution) -- [Have separate packages for CR's and instances of the custom resource](#have-separate-packages-for-crs-and-instances-of-the-custom-resource) -- [CommonLabels should be immutable](#commonlabels-should-be-immutable) - - [Resource file naming](#resource-file-naming) -- [Removing common attributes across resources](#removing-common-attributes-across-resources) - - - -## Layout package to support composability - -If your application consists of loosely coupled components e.g. backend, front-end, database consider defining these as separate kustomize packages -and then using kustomize to compose these applications into different installs e.g - -``` -components/ - /app-front - /app-backend - /app-db -installs/ - /app-standalone - /app-onprem -``` - -Defining separate packages for each component makes it easier to use composition to define new configurations; e.g. using an external database as opposed -to a database running in cluster. - -## Reuse patches - -**Note:** We are in the process of moving to Kustomize v4, see [this](https://github.com/kubeflow/manifests/issues/1797). -This method of reusing patches is outdated and will likely be replaced by -[kustomize components](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/components.md). - -We encourage reusing patches across kustomize packages when it makes sense. For example suppose we -have an onprem and standalone version of our application but both of them want to reuse -a common patch to use an external database. We could lay the packages out like so - -``` -components/ - /patches/ - /deployment-external-db.yaml -installs/ - /app-standalone - /app-onprem -``` - -The kustomization files for app-standalone could then look like the following - -``` -apiVersion: kustomize.config.k8s.io/v1beta1 -... -patchesStrategicMerge: -- ../../components/patches/deployment-external-db.yaml -``` - -### Disable security check for file outside of directory root - -To support the above layout we need to disable [kustomizes' security check](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/FAQ.md#security-file-foo-is-not-in-or-below-bar) by running with the `load_restrictor` flag: - -``` -kustomize build --load_restrictor none $target -``` - -## Command Line substitution - -To make it easy for users to override command line arguments use the following pattern. - -1. Use a config map generator to store the parameters -1. On Deployments/StatefulSets/etc... set environment variables based on the config map -1. Rely on Kubernetes to substitute environment variables into container arguments ([ref](https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config)) - -Users can then override the parameters by defining [config map overlays](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configGeneration.md). - -Using a [ConfigMapGenerator](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configGeneration.md#configmap-generation-and-rolling-updates) and including a content hash is highly prefered over not including a content hash. -Using a content hash ensures that rolling updates are triggered if the config map is changed. - -**Deprecated patterns** - -* vars should no longer be used to do command line substitution see [bit.ly/kf_kustomize_v3](https://docs.google.com/document/d/1jBayuR5YvhuGcIVAgB1F_q4NrlzUryZPyI9lCdkFTcw/edit?pli=1#heading=h.ychbuvw81fj7) - -## Eschew vars - -As noted in [kubernetes-sigs/kustomize#2052](https://github.com/kubernetes-sigs/kustomize/issues/2052) vars have a lot of downsides. -For Kubeflow in particular vars have made it difficult to compose kustomize packages because they need to be unique globally ([kubeflow/manifests#1007](https://github.com/kubeflow/manifests/issues/1007)). - -Vars should be used sparingly. Below are some guidance on acceptable use cases. - - -### Internal subsitution of fields Kustomize isn't aware of - -One ok use case for vars is getting kustomize to subsitute a value into a field kustomize wouldn't normally do substitution into. -This often happens with CRDs. For example, consider the virtual service below from [jupyter-web-app](https://github.com/kubeflow/manifests/blob/master/jupyter/jupyter-web-app/overlays/istio/virtual-service.yaml). - -``` -apiVersion: networking.istio.io/v1alpha3 -kind: VirtualService -metadata: - name: jupyter-web-app -spec: - gateways: - - kubeflow-gateway - hosts: - - '*' - http: - - ... - route: - - destination: - host: jupyter-web-app-service.$(jupyter-web-app-namespace).svc.$(clusterDomain) - port: - number: 80 -``` - -We would like kustomize to substitute namespace into the destination host. We do this by - -1. Defining a [vars](https://github.com/kubeflow/manifests/blob/393ec700e7834ca69a0832ec01ea2ecd90fb5bc4/jupyter/jupyter-web-app/base/kustomization.yaml#L63) to get the value for namespace -1. Defining a [custom configuration](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/transformerconfigs/README.md#customizing-transformer-configurations) so that the vars will be substituted into the virtual service host. - -This use of vars is acceptable because the var is internal to the kustomize package and can be given a unique enough name to prevent -conflicts when the package is composed with other applications. - -### Global substitution - -One of the most problematic use cases for vars in Kubeflow today is substituting a user supplied value into multiple applications. - -Currently we only have one use case which is substituting in cluster domain into virtual services ([ref](https://docs.google.com/document/d/1jBayuR5YvhuGcIVAgB1F_q4NrlzUryZPyI9lCdkFTcw/edit#heading=h.vyq4iltpirga)). - -We would ultimately like to get rid of the use of vars in these cases but have not settled on precise solutions. Some possible options are - -1. Using [kpt setters](https://googlecontainertools.github.io/kpt/reference/cfg/create-subst/) - - * kpt is still relatively new and we don't want to mandate/require using it - * consider adding kpt setters as appropriate so users who are willing to use kpt can avoid dealing with vars - -1. Defining custom transformers - - * e.g. we could define a new transformer for virtual services as discussed in [kubeflow/manifests#1007](https://github.com/kubeflow/manifests/issues/1007#issuecomment-599257347) - - -## Have separate packages for CR's and instances of the custom resource - -If you are adding a custom resource (e.g. CertManager) and also defining instances of those resources (e.g. ClusterIssuer) these -should be done in separate kustomize packages (see [kubeflow/manifests#1121](https://github.com/kubeflow/manifests/issues/1121)). - -Having separate packages makes it easier during deployment to ensure the custom resource is deployed and ready before trying to create instances -of the CR. - -## CommonLabels should be immutable - -As noted [here](https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html#commonlabels) commonLabels get applied to -selectors which are immutable. Therefore, commonLabels should be immutable across versions of a package to avoid causing -problems during upgrades. - -For more info see [kubeflow/manifests#1131](https://github.com/kubeflow/manifests/issues/1131) - -### Resource file naming - - Resources should be organized by kind, where the resource is in a file that is the lower-case hyphenized form of the Resource kind. For example: a Deployment would go in a file named deployment.yaml. A ClusterRoleBinding would go in a file called cluster-role-binding.yaml. If there are multiple resources within a kustomize target (eg more than one deployment), you may want to maintain a single resource per file and add a prefix|suffix of the resource name to the filename. For example the file name would be `-.yaml`. See below for an example. - -> example: /manifests/profiles - -``` -profiles -└── base - ├── README.md - ├── cluster-role-binding.yaml - ├── crd.yaml - ├── deployment.yaml - ├── kustomization.yaml - ├── role-binding.yaml - ├── role.yaml - ├── service-account.yaml - └── service.yaml -``` - -## Removing common attributes across resources - - There are often repeated attributes across resources: labels, namespace, or perhaps a common prefix used for each resource. You can move name prefixes into the kustomization.yaml file and then make adjustments within each resource; removing the prefix from its name. Additionaly you can move labels and their selectors into the kustomization.yaml. Yo can move the namespace into the kustomization.yaml. All of these will be added back into the resource by running `kustomize build`. - -> example: /manifests/profiles/base/kustomization.yaml. Contains namespace, nameprefix, commonLabels. - -``` -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- crd.yaml -- service-account.yaml -- cluster-role-binding.yaml -- role.yaml -- role-binding.yaml -- service.yaml -- deployment.yaml -namespace: kubeflow -namePrefix: profiles- -commonLabels: - kustomize.component: profiles -images: - - name: gcr.io/kubeflow-images-public/profile-controller - newName: gcr.io/kubeflow-images-public/profile-controller - newTag: v20190228-v0.4.0-rc.1-192-g1a802656-dirty-f95773 -``` - - - The original deployment in profiles looked like: - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - labels: - kustomize.component: profiles - name: profiles-deployment - namespace: kubeflow -spec: - selector: - matchLabels: - kustomize.component: profiles - template: - metadata: - labels: - kustomize.component: profiles - spec: - containers: - - command: - - /manager - image: gcr.io/kubeflow-images-public/profile-controller:v20190228-v0.4.0-rc.1-192-g1a802656-dirty-f95773 - imagePullPolicy: Always - name: manager - serviceAccountName: profiles-controller-service-account -``` - - Moving labels, namespace and the nameprefix 'profiles-' to kustomization.yaml reduces deployment.yaml to - -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: deployment -spec: - template: - spec: - containers: - - name: manager - command: - - /manager - image: gcr.io/kubeflow-images-public/profile-controller:v20190228-v0.4.0-rc.1-192-g1a802656-dirty-f95773 - imagePullPolicy: Always - serviceAccountName: controller-service-account -``` - - Note: A kustomize target should always 'build', so you should add what's needed to allow a `kustomize build` to succeed (and for unittests to work). Defining a namespace in kustomization.yaml is required to run `kustomize build`, even though there is a namespace override in the parent kustomization.yaml generated by kfctl under /manifests/profiles. This generated kustomization.yaml provides overrides using values from app.yaml and will appear within the manifest cache after running `kfctl generate...`. - From 473b1035304f847063ecaf0a44686182c437db64 Mon Sep 17 00:00:00 2001 From: Krzysztof Romanowski Date: Thu, 11 Jul 2024 14:10:21 +0200 Subject: [PATCH 10/16] Fix ml pipeline access from kfp step (#2795) * fail gh action if pipeline failed in .github/workflows/pipeline_test.yaml Signed-off-by: Krzysztof Romanowski * allow access to ml-pipeline when using trusted requestPrincipal or doesn't have auth header Signed-off-by: Krzysztof Romanowski * add more triggers for the workflow Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: Krzysztof Romanowski Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> Co-authored-by: Krzysztof Romanowski Co-authored-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- .github/workflows/pipeline_run_from_notebook.yaml | 4 ++++ .github/workflows/pipeline_test.yaml | 5 ++++- .../installs/multi-user/istio-authorization-config.yaml | 4 ++++ .../installs/multi-user/istio-authorization-config.yaml | 4 ++++ .../installs/multi-user/istio-authorization-config.yaml | 7 ++++++- 5 files changed, 22 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pipeline_run_from_notebook.yaml b/.github/workflows/pipeline_run_from_notebook.yaml index aa7e814121..52c80bd021 100644 --- a/.github/workflows/pipeline_run_from_notebook.yaml +++ b/.github/workflows/pipeline_run_from_notebook.yaml @@ -4,10 +4,14 @@ on: paths: - .github/workflows/pipeline_run_from_notebook.yaml - apps/jupyter/notebook-controller/upstream/** + - apps/pipeline/upstream/** - tests/gh-actions/kind-cluster.yaml - tests/gh-actions/install_kind.sh - tests/gh-actions/install_kustomize.sh - tests/gh-actions/install_istio.sh + - tests/gh-actions/install_cert_manager.sh + - common/cert-manager/** + - common/oidc-client/oauth2-proxy/** - common/istio*/** - common/oidc-client/** - apps/jupyter/** diff --git a/.github/workflows/pipeline_test.yaml b/.github/workflows/pipeline_test.yaml index 06b08c9aff..0fbf4b3e21 100644 --- a/.github/workflows/pipeline_test.yaml +++ b/.github/workflows/pipeline_test.yaml @@ -90,11 +90,14 @@ jobs: while True: status = client.get_run(run_id=run_id).state - if status not in ["SUCCEEDED", "FAILED", "ERROR"]: + if status in ["PENDING", "RUNNING"]: print(f"Waiting for run_id: {run_id}, status: {status}.") sleep(10) else: print(f"Run with id {run_id} finished with status: {status}.") + if status != "SUCCEEDED": + print("Pipeline failed") + raise SystemExit(1) break ' "${TOKEN}" "${KF_PROFILE}" diff --git a/apps/kfp-tekton/upstream/base/installs/multi-user/istio-authorization-config.yaml b/apps/kfp-tekton/upstream/base/installs/multi-user/istio-authorization-config.yaml index a9a45e5eb1..8b3144ad39 100644 --- a/apps/kfp-tekton/upstream/base/installs/multi-user/istio-authorization-config.yaml +++ b/apps/kfp-tekton/upstream/base/installs/multi-user/istio-authorization-config.yaml @@ -32,6 +32,10 @@ spec: - cluster.local/ns/kubeflow/sa/ml-pipeline-scheduledworkflow - cluster.local/ns/kubeflow/sa/ml-pipeline-viewer-crd-service-account - cluster.local/ns/kubeflow/sa/kubeflow-pipelines-cache + # allow access by any trusted principal + - from: + - source: + requestPrincipals: ["*"] # For user workloads, which cannot user http headers for authentication - when: - key: request.headers[kubeflow-userid] diff --git a/apps/kfp-tekton/upstream/v1/base/installs/multi-user/istio-authorization-config.yaml b/apps/kfp-tekton/upstream/v1/base/installs/multi-user/istio-authorization-config.yaml index a9a45e5eb1..8b3144ad39 100644 --- a/apps/kfp-tekton/upstream/v1/base/installs/multi-user/istio-authorization-config.yaml +++ b/apps/kfp-tekton/upstream/v1/base/installs/multi-user/istio-authorization-config.yaml @@ -32,6 +32,10 @@ spec: - cluster.local/ns/kubeflow/sa/ml-pipeline-scheduledworkflow - cluster.local/ns/kubeflow/sa/ml-pipeline-viewer-crd-service-account - cluster.local/ns/kubeflow/sa/kubeflow-pipelines-cache + # allow access by any trusted principal + - from: + - source: + requestPrincipals: ["*"] # For user workloads, which cannot user http headers for authentication - when: - key: request.headers[kubeflow-userid] diff --git a/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml b/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml index 3a08bf32c3..8b3144ad39 100644 --- a/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml +++ b/apps/pipeline/upstream/base/installs/multi-user/istio-authorization-config.yaml @@ -32,9 +32,14 @@ spec: - cluster.local/ns/kubeflow/sa/ml-pipeline-scheduledworkflow - cluster.local/ns/kubeflow/sa/ml-pipeline-viewer-crd-service-account - cluster.local/ns/kubeflow/sa/kubeflow-pipelines-cache + # allow access by any trusted principal - from: - source: - requestPrincipals: ["*"] # allow access by any trusted principal + requestPrincipals: ["*"] + # For user workloads, which cannot user http headers for authentication + - when: + - key: request.headers[kubeflow-userid] + notValues: ['*'] --- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy From 029b269e3919d85f883ce561087e51d1da9507d3 Mon Sep 17 00:00:00 2001 From: biswajit-9776 <115724497+biswajit-9776@users.noreply.github.com> Date: Mon, 22 Jul 2024 03:46:00 +0530 Subject: [PATCH 11/16] Documented about how to change default password in README.md (#2799) * Documented about how to change default password in README.md Signed-off-by: biswajit-9776 * Made requested changes Signed-off-by: biswajit-9776 --------- Signed-off-by: biswajit-9776 --- README.md | 44 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 3d0e4ca2c6..e431fb5f0d 100644 --- a/README.md +++ b/README.md @@ -465,25 +465,53 @@ If you absolutely need to expose Kubeflow over HTTP, you can disable the `Secure ### Change default user password -For security reasons, we don't want to use the default password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password before deploying. To define a password for the default user: +For security reasons, we don't want to use the default password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password and apply it either **before creating the cluster** or **after creating the cluster**. -1. Pick a password for the default user, with email `user@example.com`, and hash it using `bcrypt`: - -TODO this changed slightly in https://github.com/kubeflow/manifests/pull/2669 and https://github.com/kubeflow/manifests/pull/2229 +Pick a password for the default user, with email `user@example.com`, and hash it using `bcrypt`: ```sh python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))' ``` -2. Edit `common/dex/base/config-map.yaml` and fill the relevant field with the hash of the password you chose: +For example, running the above command locally with required packages like _passlib_ would look as follows: + ```sh + python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))' + Password: <--- Enter the password here + $2y$12$vIm8CANhuWui0J1p3jYeGeuM28Qcn76IFMaFWvZCG5ZkKZ4MjTF4u <--- GENERATED_HASH_FOR_ENTERED_PASSWORD + ``` + +#### Before creating the cluster: + +1. Edit `common/dex/base/dex-passwords.yaml` and fill the relevant field with the hash of the password you chose: ```yaml ... - staticPasswords: - - email: user@example.com - hash: + stringData: + DEX_USER_PASSWORD: ``` +#### After creating the cluster: + +1. Delete the existing secret _dex-passwords_ in auth namespace using the following command: + + ```sh + kubectl delete secret dex-passwords -n auth + ``` + +2. Create secret dex-passwords with new hash using the following command: + + ```sh + kubectl create secret generic dex-passwords --from-literal=DEX_USER_PASSWORD='REPLACE_WITH_HASH' -n auth + ``` + +3. Recreate the _dex_ pod in auth namespace using the following command: + + ```sh + kubectl delete pods --all -n auth + ``` + +4. Try to login using the new dex password. + ## Upgrading and extending For modifications and in place upgrades of the Kubeflow platform we provide a rough description for advanced users: From d5e1e012223b5b6fec7b2fd664b24a4c4b4389c5 Mon Sep 17 00:00:00 2001 From: Ricardo Martinelli de Oliveira Date: Mon, 22 Jul 2024 07:22:01 -0300 Subject: [PATCH 12/16] Update kubeflow/katib manifests from v0.17.0 (#2801) Signed-off-by: Ricardo M. Oliveira --- .../controller/trial-templates.yaml | 8 ++--- .../katib-cert-manager/katib-config.yaml | 30 +++++++++---------- .../katib-cert-manager/kustomization.yaml | 6 ++-- .../katib-external-db/katib-config.yaml | 30 +++++++++---------- .../katib-external-db/kustomization.yaml | 6 ++-- .../katib-leader-election/katib-config.yaml | 30 +++++++++---------- .../katib-openshift/katib-config.yaml | 30 +++++++++---------- .../katib-openshift/kustomization.yaml | 6 ++-- .../katib-config.yaml | 30 +++++++++---------- .../kustomization.yaml | 6 ++-- .../katib-standalone/katib-config.yaml | 30 +++++++++---------- .../katib-standalone/kustomization.yaml | 6 ++-- .../katib-with-kubeflow/kustomization.yaml | 6 ++-- 13 files changed, 112 insertions(+), 112 deletions(-) diff --git a/apps/katib/upstream/components/controller/trial-templates.yaml b/apps/katib/upstream/components/controller/trial-templates.yaml index 371d63849e..13193d2149 100644 --- a/apps/katib/upstream/components/controller/trial-templates.yaml +++ b/apps/katib/upstream/components/controller/trial-templates.yaml @@ -15,7 +15,7 @@ data: spec: containers: - name: training-container - image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0 command: - "python3" - "/opt/pytorch-mnist/mnist.py" @@ -33,7 +33,7 @@ data: spec: containers: - name: training-container - image: docker.io/kubeflowkatib/enas-cnn-cifar10-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/enas-cnn-cifar10-cpu:v0.17.0 command: - python3 - -u @@ -54,7 +54,7 @@ data: spec: containers: - name: pytorch - image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0 command: - "python3" - "/opt/pytorch-mnist/mnist.py" @@ -68,7 +68,7 @@ data: spec: containers: - name: pytorch - image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.17.0 command: - "python3" - "/opt/pytorch-mnist/mnist.py" diff --git a/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml b/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml index 419dc60ab6..08464d17ed 100644 --- a/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-cert-manager/katib-config.yaml @@ -14,40 +14,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -56,4 +56,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml b/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml index 87d217ffe9..670b72d0b6 100644 --- a/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-cert-manager/kustomization.yaml @@ -22,13 +22,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesStrategicMerge: - patches/katib-cert-injection.yaml diff --git a/apps/katib/upstream/installs/katib-external-db/katib-config.yaml b/apps/katib/upstream/installs/katib-external-db/katib-config.yaml index 7fc6860950..1773b0123f 100644 --- a/apps/katib/upstream/installs/katib-external-db/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-external-db/katib-config.yaml @@ -16,40 +16,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -58,4 +58,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-external-db/kustomization.yaml b/apps/katib/upstream/installs/katib-external-db/kustomization.yaml index 05e239a78b..e8f9a95cdd 100644 --- a/apps/katib/upstream/installs/katib-external-db/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-external-db/kustomization.yaml @@ -18,13 +18,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesStrategicMerge: - patches/db-manager.yaml # Modify katib-mysql-secrets with parameters for the DB. diff --git a/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml b/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml index 901e633adc..31d51bbf64 100644 --- a/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-leader-election/katib-config.yaml @@ -17,40 +17,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -59,4 +59,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-openshift/katib-config.yaml b/apps/katib/upstream/installs/katib-openshift/katib-config.yaml index 419dc60ab6..08464d17ed 100644 --- a/apps/katib/upstream/installs/katib-openshift/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-openshift/katib-config.yaml @@ -14,40 +14,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -56,4 +56,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-openshift/kustomization.yaml b/apps/katib/upstream/installs/katib-openshift/kustomization.yaml index af40e9ac51..f98c9638a2 100644 --- a/apps/katib/upstream/installs/katib-openshift/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-openshift/kustomization.yaml @@ -30,13 +30,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesJson6902: # Annotate Service to delegate TLS-secret generation to OpenShift service controller diff --git a/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml b/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml index 7fc6860950..1773b0123f 100644 --- a/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-standalone-postgres/katib-config.yaml @@ -16,40 +16,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -58,4 +58,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml b/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml index 48054c5f44..fa70306aff 100644 --- a/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-standalone-postgres/kustomization.yaml @@ -20,13 +20,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesJson6902: - target: group: apps diff --git a/apps/katib/upstream/installs/katib-standalone/katib-config.yaml b/apps/katib/upstream/installs/katib-standalone/katib-config.yaml index 7fc6860950..1773b0123f 100644 --- a/apps/katib/upstream/installs/katib-standalone/katib-config.yaml +++ b/apps/katib/upstream/installs/katib-standalone/katib-config.yaml @@ -16,40 +16,40 @@ init: runtime: metricsCollectors: - kind: StdOut - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: File - image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/file-metrics-collector:v0.17.0 - kind: TensorFlowEvent - image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/tfevent-metrics-collector:v0.17.0 resources: limits: memory: 1Gi suggestions: - algorithmName: random - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: tpe - image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperopt:v0.17.0 - algorithmName: grid - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: hyperband - image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-hyperband:v0.17.0 - algorithmName: bayesianoptimization - image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-skopt:v0.17.0 - algorithmName: cmaes - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: sobol - image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-goptuna:v0.17.0 - algorithmName: multivariate-tpe - image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-optuna:v0.17.0 - algorithmName: enas - image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-enas:v0.17.0 resources: limits: memory: 400Mi - algorithmName: darts - image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-darts:v0.17.0 - algorithmName: pbt - image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/suggestion-pbt:v0.17.0 persistentVolumeClaimSpec: accessModes: - ReadWriteMany @@ -58,4 +58,4 @@ runtime: storage: 5Gi earlyStoppings: - algorithmName: medianstop - image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0-rc.1 + image: docker.io/kubeflowkatib/earlystopping-medianstop:v0.17.0 diff --git a/apps/katib/upstream/installs/katib-standalone/kustomization.yaml b/apps/katib/upstream/installs/katib-standalone/kustomization.yaml index 24857c5733..146a5e6877 100644 --- a/apps/katib/upstream/installs/katib-standalone/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-standalone/kustomization.yaml @@ -20,13 +20,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 configMapGenerator: - name: katib-config behavior: create diff --git a/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml b/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml index 018c9ef43e..621651b15d 100644 --- a/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml +++ b/apps/katib/upstream/installs/katib-with-kubeflow/kustomization.yaml @@ -11,13 +11,13 @@ resources: images: - name: docker.io/kubeflowkatib/katib-controller newName: docker.io/kubeflowkatib/katib-controller - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-db-manager newName: docker.io/kubeflowkatib/katib-db-manager - newTag: v0.17.0-rc.1 + newTag: v0.17.0 - name: docker.io/kubeflowkatib/katib-ui newName: docker.io/kubeflowkatib/katib-ui - newTag: v0.17.0-rc.1 + newTag: v0.17.0 patchesStrategicMerge: - patches/remove-namespace.yaml From 858107b90a54987765acb68a889301f40b8f972b Mon Sep 17 00:00:00 2001 From: Ricardo Martinelli de Oliveira Date: Mon, 22 Jul 2024 07:23:00 -0300 Subject: [PATCH 13/16] Update kubeflow/training-operator manifests from v1.8.0 (#2802) Signed-off-by: Ricardo M. Oliveira --- README.md | 2 +- .../upstream/overlays/kubeflow/kustomization.yaml | 2 +- .../upstream/overlays/standalone/kustomization.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e431fb5f0d..d9131fbda0 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ This repo periodically syncs all official Kubeflow components from their respect | Component | Local Manifests Path | Upstream Revision | | - | - | - | -| Training Operator | apps/training-operator/upstream | [v1.8.0-rc.1](https://github.com/kubeflow/training-operator/tree/v1.8.0-rc.1/manifests) | +| Training Operator | apps/training-operator/upstream | [v1.8.0](https://github.com/kubeflow/training-operator/tree/v1.8.0/manifests) | | Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/notebook-controller/config) | | PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/pvcviewer-controller/config) | | Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/tensorboard-controller/config) | diff --git a/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml b/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml index 71ce5ef7b5..f4dabc273e 100644 --- a/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml +++ b/apps/training-operator/upstream/overlays/kubeflow/kustomization.yaml @@ -6,7 +6,7 @@ resources: - kubeflow-training-roles.yaml images: - name: kubeflow/training-operator - newTag: "v1-4485b0a" + newTag: "v1-9e52eb7" # TODO (tenzen-y): Once we support cert-manager, we need to remove this secret generation. # REF: https://github.com/kubeflow/training-operator/issues/2049 secretGenerator: diff --git a/apps/training-operator/upstream/overlays/standalone/kustomization.yaml b/apps/training-operator/upstream/overlays/standalone/kustomization.yaml index 0ecb165a23..0a6eb6ffbe 100644 --- a/apps/training-operator/upstream/overlays/standalone/kustomization.yaml +++ b/apps/training-operator/upstream/overlays/standalone/kustomization.yaml @@ -6,7 +6,7 @@ resources: - namespace.yaml images: - name: kubeflow/training-operator - newTag: "v1-4485b0a" + newTag: "v1-9e52eb7" secretGenerator: - name: training-operator-webhook-cert options: From c29b4deb3f08241626276a645c9ff99e0521aabd Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 22 Jul 2024 12:42:01 +0200 Subject: [PATCH 14/16] Synchronize training operator and katib manifests fixes (#2806) * fix the version numbers in the scripts Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * fix the katib directory Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> * Update kubeflow/katib manifests from v0.17.0 Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --------- Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 2 +- hack/synchronize-katib-manifests.sh | 6 +++--- hack/synchronize-training-operator-manifests.sh | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index d9131fbda0..d686670719 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ This repo periodically syncs all official Kubeflow components from their respect | Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/jupyter/manifests) | | Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/tensorboards/manifests) | | Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/volumes/manifests) | -| Katib | apps/katib/upstream | [v0.17.0-rc.0](https://github.com/kubeflow/katib/tree/v0.17.0-rc.0/manifests/v1beta1) | +| Katib | apps/katib/upstream | [v0.17.0](https://github.com/kubeflow/katib/tree/v0.17.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | | KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | | Kubeflow Pipelines | apps/pipeline/upstream | [2.2.0](https://github.com/kubeflow/pipelines/tree/2.2.0/manifests/kustomize) | diff --git a/hack/synchronize-katib-manifests.sh b/hack/synchronize-katib-manifests.sh index d1cd5409c8..1edcdbaaa1 100644 --- a/hack/synchronize-katib-manifests.sh +++ b/hack/synchronize-katib-manifests.sh @@ -15,7 +15,7 @@ set -euxo pipefail IFS=$'\n\t' -COMMIT="v0.17.0-rc.0" # You can use tags as well +COMMIT="v0.17.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-katib} BRANCH=${BRANCH:=synchronize-kubeflow-katib-manifests-${COMMIT?}} @@ -69,8 +69,8 @@ cp $SRC_DIR/katib/manifests/v1beta1 $DST_DIR -r echo "Successfully copied all manifests." echo "Updating README..." -SRC_TXT="\[.*\](https://github.com/kubeflow/katib/tree/.*/manifests/kustomize)" -DST_TXT="\[$COMMIT\](https://github.com/kubeflow/katib/tree/$COMMIT/manifests/kustomize)" +SRC_TXT="\[.*\](https://github.com/kubeflow/katib/tree/.*/manifests/v1beta1)" +DST_TXT="\[$COMMIT\](https://github.com/kubeflow/katib/tree/$COMMIT/manifests/v1beta1)" sed -i "s|$SRC_TXT|$DST_TXT|g" ${MANIFESTS_DIR}/README.md diff --git a/hack/synchronize-training-operator-manifests.sh b/hack/synchronize-training-operator-manifests.sh index 49f69080fb..4ea9aafb48 100644 --- a/hack/synchronize-training-operator-manifests.sh +++ b/hack/synchronize-training-operator-manifests.sh @@ -15,7 +15,7 @@ set -euxo pipefail IFS=$'\n\t' -COMMIT="v1.8.0-rc.1" # You can use tags as well +COMMIT="v1.8.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-training-operator} BRANCH=${BRANCH:=synchronize-kubeflow-training-operator-manifests-${COMMIT?}} From 43a0e6ad69975c60a81d54deddeecd851b8eb808 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 22 Jul 2024 13:22:01 +0200 Subject: [PATCH 15/16] Update kubeflow/kubeflow manifests from v1.9.0 (#2807) fix tags fix tags fix tags fix tags fix tags Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 18 +++++++++--------- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- apps/profiles/upstream/base/kustomization.yaml | 2 +- .../overlays/kubeflow/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- .../upstream/base/kustomization.yaml | 2 +- hack/synchronize-kubeflow-manifests.sh | 2 +- 12 files changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index d686670719..b928b389f5 100644 --- a/README.md +++ b/README.md @@ -44,15 +44,15 @@ This repo periodically syncs all official Kubeflow components from their respect | Component | Local Manifests Path | Upstream Revision | | - | - | - | | Training Operator | apps/training-operator/upstream | [v1.8.0](https://github.com/kubeflow/training-operator/tree/v1.8.0/manifests) | -| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/notebook-controller/config) | -| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/pvcviewer-controller/config) | -| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/tensorboard-controller/config) | -| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/centraldashboard/manifests) | -| Profiles + KFAM | apps/profiles/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/profile-controller/config) | -| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/admission-webhook/manifests) | -| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/jupyter/manifests) | -| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/tensorboards/manifests) | -| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0-rc.2](https://github.com/kubeflow/kubeflow/tree/v1.9.0-rc.2/components/crud-web-apps/volumes/manifests) | +| Notebook Controller | apps/jupyter/notebook-controller/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/notebook-controller/config) | +| PVC Viewer Controller | apps/pvcviewer-roller/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/pvcviewer-controller/config) | +| Tensorboard Controller | apps/tensorboard/tensorboard-controller/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/tensorboard-controller/config) | +| Central Dashboard | apps/centraldashboard/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/centraldashboard/manifests) | +| Profiles + KFAM | apps/profiles/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/profile-controller/config) | +| PodDefaults Webhook | apps/admission-webhook/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/admission-webhook/manifests) | +| Jupyter Web App | apps/jupyter/jupyter-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/jupyter/manifests) | +| Tensorboards Web App | apps/tensorboard/tensorboards-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/tensorboards/manifests) | +| Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/volumes/manifests) | | Katib | apps/katib/upstream | [v0.17.0](https://github.com/kubeflow/katib/tree/v0.17.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | | KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | diff --git a/apps/admission-webhook/upstream/base/kustomization.yaml b/apps/admission-webhook/upstream/base/kustomization.yaml index 2b70cdaeb4..04fe85de8c 100644 --- a/apps/admission-webhook/upstream/base/kustomization.yaml +++ b/apps/admission-webhook/upstream/base/kustomization.yaml @@ -16,7 +16,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/poddefaults-webhook newName: docker.io/kubeflownotebookswg/poddefaults-webhook - newTag: v1.9.0-rc.2 + newTag: v1.9.0 namespace: kubeflow generatorOptions: disableNameSuffixHash: true diff --git a/apps/centraldashboard/upstream/base/kustomization.yaml b/apps/centraldashboard/upstream/base/kustomization.yaml index 99cdab21b1..08f70ab264 100644 --- a/apps/centraldashboard/upstream/base/kustomization.yaml +++ b/apps/centraldashboard/upstream/base/kustomization.yaml @@ -13,7 +13,7 @@ resources: images: - name: docker.io/kubeflownotebookswg/centraldashboard newName: docker.io/kubeflownotebookswg/centraldashboard - newTag: v1.9.0-rc.2 + newTag: v1.9.0 configMapGenerator: - envs: - params.env diff --git a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml index c03654976b..ec62bdd747 100644 --- a/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml +++ b/apps/jupyter/jupyter-web-app/upstream/base/kustomization.yaml @@ -23,7 +23,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/jupyter-web-app newName: docker.io/kubeflownotebookswg/jupyter-web-app - newTag: v1.9.0-rc.2 + newTag: v1.9.0 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml index e1d4830ea2..a6b02fd248 100644 --- a/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml +++ b/apps/jupyter/notebook-controller/upstream/base/kustomization.yaml @@ -5,4 +5,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/notebook-controller newName: docker.io/kubeflownotebookswg/notebook-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/profiles/upstream/base/kustomization.yaml b/apps/profiles/upstream/base/kustomization.yaml index 0d29098f03..c6a15a0bf9 100644 --- a/apps/profiles/upstream/base/kustomization.yaml +++ b/apps/profiles/upstream/base/kustomization.yaml @@ -12,7 +12,7 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/profile-controller newName: docker.io/kubeflownotebookswg/profile-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 configMapGenerator: - name: namespace-labels-data diff --git a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml index e68ee70492..1b8fba691c 100644 --- a/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml +++ b/apps/profiles/upstream/overlays/kubeflow/kustomization.yaml @@ -29,4 +29,4 @@ vars: images: - name: docker.io/kubeflownotebookswg/kfam newName: docker.io/kubeflownotebookswg/kfam - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/pvcviewer-controller/upstream/base/kustomization.yaml b/apps/pvcviewer-controller/upstream/base/kustomization.yaml index 4eb0ae99ac..259b701ef7 100644 --- a/apps/pvcviewer-controller/upstream/base/kustomization.yaml +++ b/apps/pvcviewer-controller/upstream/base/kustomization.yaml @@ -6,4 +6,4 @@ resources: images: - name: docker.io/kubeflownotebookswg/pvcviewer-controller newName: docker.io/kubeflownotebookswg/pvcviewer-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml index c16af71f59..0a651adf84 100644 --- a/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboard-controller/upstream/base/kustomization.yaml @@ -14,4 +14,4 @@ patchesStrategicMerge: images: - name: docker.io/kubeflownotebookswg/tensorboard-controller newName: docker.io/kubeflownotebookswg/tensorboard-controller - newTag: v1.9.0-rc.2 + newTag: v1.9.0 diff --git a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml index 3d6596f1f3..366638beec 100644 --- a/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml +++ b/apps/tensorboard/tensorboards-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/tensorboards-web-app newName: docker.io/kubeflownotebookswg/tensorboards-web-app - newTag: v1.9.0-rc.2 + newTag: v1.9.0 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/apps/volumes-web-app/upstream/base/kustomization.yaml b/apps/volumes-web-app/upstream/base/kustomization.yaml index c2343baa1c..e81cdf755a 100644 --- a/apps/volumes-web-app/upstream/base/kustomization.yaml +++ b/apps/volumes-web-app/upstream/base/kustomization.yaml @@ -14,7 +14,7 @@ commonLabels: images: - name: docker.io/kubeflownotebookswg/volumes-web-app newName: docker.io/kubeflownotebookswg/volumes-web-app - newTag: v1.9.0-rc.2 + newTag: v1.9.0 # We need the name to be unique without the suffix because the original name is what # gets used with patches configMapGenerator: diff --git a/hack/synchronize-kubeflow-manifests.sh b/hack/synchronize-kubeflow-manifests.sh index 96c5154f47..71bbe3e4a8 100644 --- a/hack/synchronize-kubeflow-manifests.sh +++ b/hack/synchronize-kubeflow-manifests.sh @@ -14,7 +14,7 @@ set -euo pipefail IFS=$'\n\t' -COMMIT="v1.9.0-rc.2" # You can use tags as well +COMMIT="v1.9.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kubeflow-kubeflow} BRANCH=${BRANCH:=synchronize-kubeflow-kubeflow-manifests-${COMMIT?}} From a38c2be88fbafb0844c0231f0062e4b3719d4737 Mon Sep 17 00:00:00 2001 From: Julius von Kohout <45896133+juliusvonkohout@users.noreply.github.com> Date: Mon, 22 Jul 2024 13:30:01 +0200 Subject: [PATCH 16/16] Update kserve models web application manifests from 0.13.0 (#2808) fix tags Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com> --- README.md | 2 +- hack/synchronize-kserve-web-app-manifests.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b928b389f5..7b85df7d35 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ This repo periodically syncs all official Kubeflow components from their respect | Volumes Web App | apps/volumes-web-app/upstream | [v1.9.0](https://github.com/kubeflow/kubeflow/tree/v1.9.0/components/crud-web-apps/volumes/manifests) | | Katib | apps/katib/upstream | [v0.17.0](https://github.com/kubeflow/katib/tree/v0.17.0/manifests/v1beta1) | | KServe | contrib/kserve/kserve | [0.13.0](https://github.com/kserve/kserve/releases/tag/v0.13.0) | -| KServe Models Web App | contrib/kserve/models-web-app | [0.13.0-rc.0](https://github.com/kserve/models-web-app/tree/0.13.0-rc.0/config) | +| KServe Models Web App | contrib/kserve/models-web-app | [0.13.0](https://github.com/kserve/models-web-app/tree/0.13.0/config) | | Kubeflow Pipelines | apps/pipeline/upstream | [2.2.0](https://github.com/kubeflow/pipelines/tree/2.2.0/manifests/kustomize) | | Kubeflow Tekton Pipelines | apps/kfp-tekton/upstream | [2.0.5](https://github.com/kubeflow/kfp-tekton/tree/2.0.5/manifests/kustomize) | | Kubeflow Model Registry | apps/model-registry/upstream | [v0.2.1-alpha](https://github.com/kubeflow/model-registry/tree/v0.2.1-alpha/manifests/kustomize) | diff --git a/hack/synchronize-kserve-web-app-manifests.sh b/hack/synchronize-kserve-web-app-manifests.sh index beabded784..20954bd3cb 100644 --- a/hack/synchronize-kserve-web-app-manifests.sh +++ b/hack/synchronize-kserve-web-app-manifests.sh @@ -11,7 +11,7 @@ # repository, based on that local branch -COMMIT="0.13.0-rc.0" # You can use tags as well +COMMIT="0.13.0" # You can use tags as well SRC_DIR=${SRC_DIR:=/tmp/kserve-models-web-app} BRANCH=${BRANCH:=synchronize-kserve-web-app-manifests-${COMMIT?}}