Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debugging warnings in pss workflow #2866

Merged
merged 79 commits into from
Sep 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
7ab6e55
Debugging warnings in pss workflow
biswajit-9776 Sep 2, 2024
0f00b27
Shifted order of applying patches to workflow
biswajit-9776 Sep 4, 2024
a3fd0ed
Fixed linting
biswajit-9776 Sep 4, 2024
b239d2a
Increased timeout
biswajit-9776 Sep 4, 2024
13f5972
Skip patching dynamic namespaces
biswajit-9776 Sep 4, 2024
66f6493
Remove debugging job
biswajit-9776 Sep 4, 2024
e8829c3
Debugging by restarting deployments
biswajit-9776 Sep 4, 2024
b22ced2
Removed redundant line from patch
biswajit-9776 Sep 4, 2024
ab29ecf
Replace restart with wait command
biswajit-9776 Sep 4, 2024
39bd95b
Replace wait for all command with individual wait command
biswajit-9776 Sep 4, 2024
5f0741a
Added wait command for pods in kubeflow namespace
biswajit-9776 Sep 4, 2024
cbed790
Fixed linting
biswajit-9776 Sep 4, 2024
e00b5da
Separated wait commands for separate namespaces
biswajit-9776 Sep 4, 2024
519fef4
Increased timeout to 600s
biswajit-9776 Sep 4, 2024
928ffb9
Added wait commands in loop
biswajit-9776 Sep 5, 2024
b504f36
Fixed typo
biswajit-9776 Sep 5, 2024
363113c
Debugging failing wait commands
biswajit-9776 Sep 5, 2024
63821aa
Added log process in background
biswajit-9776 Sep 6, 2024
c3a7798
Fixed type
biswajit-9776 Sep 6, 2024
262c78c
Fixed typo
biswajit-9776 Sep 6, 2024
9a21438
Added describe command
biswajit-9776 Sep 6, 2024
86c9577
Added runAsUser to debug
biswajit-9776 Sep 6, 2024
3db6583
Added UID 1000 to all pods
biswajit-9776 Sep 6, 2024
74eb10e
Added single wait command
biswajit-9776 Sep 6, 2024
9ace6ac
Debugging
biswajit-9776 Sep 6, 2024
005500c
Removed background process
biswajit-9776 Sep 6, 2024
77f400a
Increased timeout
biswajit-9776 Sep 6, 2024
6f9d201
Retesting
biswajit-9776 Sep 6, 2024
241f948
Removed UID from profile controller
biswajit-9776 Sep 6, 2024
a564ded
Debugging profiles-controller
biswajit-9776 Sep 6, 2024
3de18f4
Increased timeout for profiles-controller
biswajit-9776 Sep 6, 2024
b7dd0bf
Pausing patch to profile-controller
biswajit-9776 Sep 6, 2024
c5ccd37
Fixed indentation
biswajit-9776 Sep 6, 2024
94a351f
Fixed error
biswajit-9776 Sep 6, 2024
17a35c4
Fixed error
biswajit-9776 Sep 6, 2024
7f521f2
Fixed error
biswajit-9776 Sep 6, 2024
04fb9a7
Debugging cache-server
biswajit-9776 Sep 7, 2024
09e0c84
Debugging cache-server
biswajit-9776 Sep 7, 2024
7492290
Increased timeout for cache-server
biswajit-9776 Sep 7, 2024
bd13b6d
Debugging
biswajit-9776 Sep 7, 2024
5cc7aa6
Deleting pods exclusively
biswajit-9776 Sep 7, 2024
43f6b3b
Added GID
biswajit-9776 Sep 9, 2024
a495be7
Debugging cache-server
biswajit-9776 Sep 9, 2024
f496072
Sleeping for 300s
biswajit-9776 Sep 9, 2024
523febc
Updated GID to 1000
biswajit-9776 Sep 9, 2024
cdbea53
Added GID to all pods
biswajit-9776 Sep 9, 2024
cb04ee9
Checking pods securityContext
biswajit-9776 Sep 9, 2024
8448450
Debugging
biswajit-9776 Sep 9, 2024
233dc59
Debugging without IDs
biswajit-9776 Sep 9, 2024
4bbbdeb
Debugging
biswajit-9776 Sep 11, 2024
9d8d641
Debugging
biswajit-9776 Sep 11, 2024
f31179a
Debugging
biswajit-9776 Sep 12, 2024
c4693b1
Added GID
biswajit-9776 Sep 12, 2024
d9b6250
Debugging
biswajit-9776 Sep 12, 2024
5a2f711
Debugging
biswajit-9776 Sep 12, 2024
c59a8f8
Debugging
biswajit-9776 Sep 12, 2024
8754c92
Debugging
biswajit-9776 Sep 12, 2024
5e8fd14
Debugging
biswajit-9776 Sep 12, 2024
10a23ae
Debugging
biswajit-9776 Sep 12, 2024
bce60a3
Debugging
biswajit-9776 Sep 14, 2024
9b95dc6
Debugging
biswajit-9776 Sep 14, 2024
a713a4c
Debugging
biswajit-9776 Sep 14, 2024
bb20a6b
Debugging
biswajit-9776 Sep 14, 2024
c2af245
Debugging
biswajit-9776 Sep 14, 2024
d6ebc3f
Debugging
biswajit-9776 Sep 15, 2024
86c372c
Debugging
biswajit-9776 Sep 15, 2024
b88fe8f
Added wait time for istio-cni test
biswajit-9776 Sep 15, 2024
523f9a0
Debugging
biswajit-9776 Sep 15, 2024
2bd7e07
Increased timeout
biswajit-9776 Sep 15, 2024
d0f3cc6
Added dynamic ns to pss_test
biswajit-9776 Sep 15, 2024
5743a1f
Fix lint
biswajit-9776 Sep 15, 2024
65ec9f7
Configured istio initContainer
biswajit-9776 Sep 16, 2024
b00fe48
Fixed lint
biswajit-9776 Sep 16, 2024
0cd0454
Added seccompProfile attribute for istio-proxy
biswajit-9776 Sep 16, 2024
0cd2870
Adding PSS label to dynamic ns
biswajit-9776 Sep 16, 2024
f3cace5
Reordered dynamic ns test
biswajit-9776 Sep 16, 2024
2cb79fb
Commented the dynamic ns tests
biswajit-9776 Sep 16, 2024
5ed56cf
Fix lint
biswajit-9776 Sep 16, 2024
e40311f
Removing debugging stuffs
biswajit-9776 Sep 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 30 additions & 31 deletions .github/workflows/pss_test.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Apply PSS labels to namespaces
on:

Check warning on line 2 in .github/workflows/pss_test.yaml

View workflow job for this annotation

GitHub Actions / format_YAML_files

2:1 [truthy] truthy value should be one of [false, true]
pull_request:
paths:
- tests/gh-actions/install_KinD_create_KinD_cluster_install_kustomize.sh
Expand All @@ -11,7 +11,7 @@
- common/cert-manager/**
- common/oauth2-proxy/**
- common/istio*/**
- tests/gh-actions/install_istio_with_ext_auth.sh
- tests/gh-actions/install_istio-cni.sh
- tests/gh-actions/install_multitenancy.sh

jobs:
Expand All @@ -27,12 +27,25 @@
- name: Install kubectl
run: ./tests/gh-actions/install_kubectl.sh

- name: Install all deployments from static namespaces
- name: Install all istio-cni resources and kubeflow namespace
run: |
kustomize build common/kubeflow-namespace/base | kubectl apply -f -
./tests/gh-actions/install_cert_manager.sh
./tests/gh-actions/install_istio_with_ext_auth.sh
kustomize build common/istio-1-22/kubeflow-istio-resources/base | kubectl apply -f -
./tests/gh-actions/install_istio-cni.sh
kustomize build common/istio-cni-1-22/kubeflow-istio-resources/base | kubectl apply -f -

- name: Configure istio init container with seccompProfile attribute
run: |
kubectl get cm istio-sidecar-injector -n istio-system -o yaml > temporary_patch.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good :-) according to the workflow results it seems to work, but please verify it manually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I did run it manually and opened up the configmap to observe the changes in place as we want. Also, I have removed the comments which I can later add as commits for dynamic namespace.

Copy link
Contributor Author

@biswajit-9776 biswajit-9776 Sep 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ml-pipeline pod in my local system does seem to fail with CrashLoopBackOff as I open it up. Maybe I need to increase my fs.inotify.max_user_{instances, watches} .

sed -i '0,/runAsNonRoot: true/{s//&\n seccompProfile:\n type: RuntimeDefault/}' temporary_patch.yaml
sed -i '/runAsNonRoot: true/{N; /runAsUser: {{ .ProxyUID | default "1337" }}/a\
seccompProfile:\n type: RuntimeDefault
}' temporary_patch.yaml
kubectl apply -f temporary_patch.yaml
rm temporary_patch.yaml

- name: Install all other deployments of static namespaces
run: |
./tests/gh-actions/install_multi_tenancy.sh
kustomize build ./common/oauth2-proxy/overlays/m2m-self-signed | kubectl apply -f -
echo "Waiting for all oauth2-proxy pods to become ready..."
Expand All @@ -44,33 +57,6 @@
- name: Install KF Pipelines
run: ./tests/gh-actions/install_pipelines.sh

- name: Apply Pod Security Standards baseline levels for static namespaces
run: ./tests/gh-actions/enable_baseline_PSS.sh

- name: Apply Pod Security Standards baseline levels for dynamic namespaces
run: |
cat << EOF > ./kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- apps/profiles/upstream/overlays/kubeflow
components:
- contrib/security/PSS/dynamic/baseline
EOF
kubectl apply -k .
rm ./kustomization.yaml
kubectl -n kubeflow wait --for=condition=Ready pods -l kustomize.component=profiles --timeout 180s

- name: Unapply applied baseline values
run: |
NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow")
for NAMESPACE in "${NAMESPACES[@]}"; do
if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
fi
done
sleep 10

- name: Apply patches to clear warnings
run: |
DIRECTORY="contrib/security/PSS/patches"
Expand All @@ -87,6 +73,19 @@
kubectl patch "$KIND" "$NAME" -n "$NAMESPACE" --patch-file "$file"
fi
done
sleep 300

- name: Apply Pod Security Standards baseline levels for static namespaces
run: ./tests/gh-actions/enable_baseline_PSS.sh

- name: Unapply applied baseline labels
run: |
NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow")
for NAMESPACE in "${NAMESPACES[@]}"; do
if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
fi
done

- name: Applying Pod Security Standards restricted levels for static namespaces
run: ./tests/gh-actions/enable_restricted_PSS.sh
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/cache-server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/kfam.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/manager.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
13 changes: 13 additions & 0 deletions contrib/security/PSS/patches/metacontroller.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: metacontroller
namespace: kubeflow
spec:
template:
spec:
containers:
- name: metacontroller
securityContext:
seccompProfile:
type: RuntimeDefault
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/metadata-envoy-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/metadata-grpc-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/metadata-writer.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/minio.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/ml-pipeline-ui.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/ml-pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
2 changes: 2 additions & 0 deletions contrib/security/PSS/patches/mysql.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ spec:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 0
capabilities:
drop:
- ALL
1 change: 0 additions & 1 deletion contrib/security/PSS/patches/oauth2-proxy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ metadata:
name: oauth2-proxy
namespace: oauth2-proxy
spec:
replicas: 2
template:
spec:
containers:
Expand Down
14 changes: 14 additions & 0 deletions contrib/security/PSS/patches/workflow-controller.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: workflow-controller
namespace: kubeflow
spec:
template:
spec:
containers:
- name: workflow-controller
securityContext:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
5 changes: 4 additions & 1 deletion tests/gh-actions/install_istio-cni.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,7 @@ echo "Installing Istio-cni ..."
cd common/istio-cni-1-22
kustomize build istio-crds/base | kubectl apply -f -
kustomize build istio-namespace/base | kubectl apply -f -
kustomize build istio-install/base | kubectl apply -f -
kustomize build istio-install/base | kubectl apply -f -

echo "Waiting for all Istio Pods to become ready..."
kubectl wait --for=condition=Ready pods --all -n istio-system --timeout 300s
Loading