Updated canary workshop (#33)

* fleet create * GitOps Secure Build * added ips * Secure Build: imdb * Secure Build: imdb * GitOps Secure Build * added flagger app * Secure Build: flagger * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * added flagger-grafana app * Secure Build: flagger-grafana * GitOps Secure Build * Secure Build: flagger-grafana * GitOps Secure Build * update proxy * GitOps Secure Build * updated app * GitOps Secure Build * Secure Build: flagger-grafana * GitOps Secure Build * Secure Build: flagger-grafana * GitOps Secure Build * updated app * updated apps * updated app * GitOps Secure Build * updated ap * GitOps Secure Build * updated apps * GitOps Secure Build * updated apps * updated apps * GitOps Secure Build * updated apps * GitOps Secure Build * updated apps * GitOps Secure Build * updated app * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * build imdb * GitOps Secure Build * build imdb * GitOps Secure Build * build imdb * GitOps Secure Build * update * GitOps Secure Build * Secure Build: flagger-grafana * GitOps Secure Build * Secure Build: flagger * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * cleanup * updated docs * fleet create * GitOps Secure Build * added ips * added flagger * Secure Build: flagger-grafana * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * Secure Build: imdb * GitOps Secure Build * updated docs * updated docs * deleted cluster * cleanup * updated docs * updated docs * updated docs * Update README.md Co-authored-by: pib-gitops <pib-gitops@outlook.com>
kubernetes101 · Nov 7, 2022 · e2a7572 · e2a7572
1 parent e1089d3
commit e2a7572
Show file tree

Hide file tree

Showing 9 changed files with 145 additions and 67 deletions.
diff --git a/labs/advanced-labs/canary/README.md b/labs/advanced-labs/canary/README.md
@@ -21,20 +21,22 @@ git branch --show-current
 
 ```
 
-## Install Flagger
+## Install Flagger and Grafana
+
+> Note: Grafana installation is optional. It provides visibility in to primary and canary workloads during canary analysis.
 
 ```bash
 
-# make sure you are in canary directory
-cd $PIB_BASE/labs/advanced-labs/canary
+# make sure you are in the root directory
+cd $PIB_BASE
 
-# copy flagger to apps directory
-cp -R ./flagger ../../../apps
+# copy flagger and flagger-grafana to apps directory
+cp -R ./labs/advanced-labs/canary/flagger apps
+cp -R ./labs/advanced-labs/canary/flagger-grafana apps
 
 # add and commit the flagger app
-cd $PIB_BASE
 git add .
-git commit -am "added flagger app"
+git commit -am "added flagger and grafana apps"
 git push
 
 cd apps/flagger
@@ -43,7 +45,18 @@ cd apps/flagger
 flt targets list
 
 # clear the targets if not []
-flt targets clear
+# flt targets clear
+
+# add all clusters as a target
+flt targets add all
+
+cd ../flagger-grafana
+
+# check deploy targets (should be [])
+flt targets list
+
+# clear the targets if not []
+# flt targets clear
 
 # add all clusters as a target
 flt targets add all
@@ -60,39 +73,44 @@ flt targets deploy
 
 ### Check deployment
 
-- Once the action completes successfully
+Once the action completes successfully you should see flagger added to your cluster
 
 ```bash
 
-# you should see flagger added to your cluster
-git pull
-
-# force flux to sync
+# pull the latest and force flux to sync
 # flux will sync on a schedule - this command forces it to sync now for debugging
+git pull
 flt sync
 
 # check that flagger is deployed to your cluster
 # NOTE: We also deploy prometheus to scrape metrics to monitor Canary deployment
 flt check app flagger
 flt check app prometheus
+flt check app flagger-grafana
 
 ```
 
 ## Update reference app to use Canary deployment Strategy
 
-- To update IMDb reference app to use canary deployment template:
-  - Update `apps/imdb/app.yaml` with template value </br>
-      `template: pib-service-canary`
+- Update IMDb reference app to use canary deployment template:
 
     ```bash
 
+    cd $PIB_BASE
+
+    # update imdb app.yaml with canary template
+    sed -i 's,pib-service,pib-service-canary,g' apps/imdb/app.yaml
+
+    # validate app.yaml
+    cat apps/imdb/app.yaml
+
     # deploy imdb with canary template
-    cd ../imdb
+    cd apps/imdb
     flt targets deploy
 
     ```
 
-  - Once the [github action](https://github.com/kubernetes101/pib-dev/actions) is completed, force flux to sync:
+- Once the [github action](https://github.com/kubernetes101/pib-dev/actions) is completed, force flux to sync:
 
      ```bash
 
@@ -103,6 +121,15 @@ flt check app prometheus
 
     ```
 
+- Validate primary and canary objects in the cluster:
+
+    ```bash
+
+    flt exec kic pods
+    flt exec kic svc
+
+    ```
+
     The reference app should be updated with Canary Deployment objects listed:
 
     ```bash
@@ -118,35 +145,28 @@ flt check app prometheus
 
     ```
 
-  - Validate primary and canary objects in the cluster:
-
-    ```bash
-
-    flt ssh $MY_CLUSTER
-    kic pods
-    kic svc
-    kubectl get canary -n imdb
-
-    # exit from cluster
-    exit
-
-    ```
 
 ## Observe automated canary promotion
 
 - Trigger a canary deployment by updating the container image for IMDb:
-  - Update `apps/imdb/app.yaml` with image tag from `latest` to `beta` </br>
-      `image: ghcr.io/cse-labs/pib-imdb:beta`
 
     ```bash
 
-    # deploy imdb with updated version
-    cd ../imdb
+    # make sure you're in imdb dir
+    cd $PIB_BASE/apps/imdb
+
+    # Update imdb app.yaml with image tag from latest to beta
+    sed -i 's,ghcr.io/cse-labs/pib-imdb:latest,ghcr.io/cse-labs/pib-imdb:beta,g' app.yaml
+
+    # validate app.yaml
+    cat app.yaml
+
+    # deploy imdb with new image
     flt targets deploy
 
     ```
 
-  - Once the [github action](https://github.com/kubernetes101/pib-dev/actions) is completed, force flux to sync:
+- Once the [github action](https://github.com/kubernetes101/pib-dev/actions) is completed, force flux to sync:
 
     ```bash
 
@@ -178,44 +198,84 @@ flt check app prometheus
   - Exit K9s: `:q <enter>`
   - Exit from cluster: `exit <enter>`
 
-## Monitoring Canary deployments using Grafana
+- Observe canary analysis in Grafana:
+  - Navigate to grafana dashboard by appending `/grafana` to the host url in the browser tab.
+  - Grafana login info
+    - admin
+    - change-me
+  - Navigate to `Envoy Canary` dashboard
 
-Flagger comes with a Grafana dashboard made for canary analysis. Install Grafana
+  ![Canary Dashboard](../../images/envoyCanaryPromotion.png)
 
-  ```bash
+## Observe automated rollback
 
-  # cd to canary directory
-  cd $PIB_BASE/labs/advanced-labs/canary
+- Trigger another canary deployment by updating the container image for IMDb:
+  > NOTE: Image tag `pib-imdb:bad` is designed to generate HTTP 500 errors to test if Flagger pauses the rollout
 
-  # copy flagger to apps directory
-  cp -R ./flagger-grafana ../../../apps
+    ```bash
 
-  # add and commit the flagger-grafana app
-  cd $PIB_BASE
-  git add .
-  git commit -am "added flagger-grafana app"
-  git push
+    # make sure you're in imdb dir
+    cd $PIB_BASE/apps/imdb
 
-  cd apps/flagger-grafana
+    # update imdb app.yaml with image tag from beta to bad
+    sed -i 's,ghcr.io/cse-labs/pib-imdb:beta,ghcr.io/cse-labs/pib-imdb:bad,g' app.yaml
 
-  # check deploy targets (should be [])
-  flt targets list
+    # validate app.yaml
+    cat app.yaml
 
-  # clear the targets if not []
-  flt targets clear
+    # deploy imdb with new image
+    flt targets deploy
+
+    ```
 
-  # add all clusters as a target
-  flt targets add all
+- Once the [github action](https://github.com/kubernetes101/pib-dev/actions) is completed, force flux to sync:
+
+    ```bash
 
-  # deploy the changes
-  flt targets deploy
+    # force flux to sync
+    # flux will sync on a schedule - this command forces it to sync now for debugging
+    git pull
+    flt sync
+
+    ```
+
+- Observe canary rollback in k9s:
+
+  ```bash
+
+  # start k9s for the cluster
+  flt ssh $MY_CLUSTER
+  k9s <enter>
 
   ```
 
-Once the [github action](https://github.com/kubernetes101/pib-dev/actions) is completed and flux sync is performed, navigate to grafana dashboard by appending `/grafana` to the host url in the browser tab.
+  - Type `:canaries <enter>` to view canary object
+  - Observe `status` and `weight` for canary promotion
+
+    > - Flagger detects the deployment version change and starts a new rollout with 20% traffic progression
+    > - When the number of failed checks reaches the canary analysis threshold casued by HTTP 500 errors, the traffic is routed back to the primary, the canary weight is scaled to zero and the rollout is marked as `failed`
+
+  - Press `enter` again and scroll to bottom to see events
+  - Press `escape` to go back
+  - Exit K9s: `:q <enter>`
+  - Exit from cluster: `exit <enter>`
+
+- Observe canary analysis in Grafana:
+
+  ![Canary Dashboard](../../images/envoyCanaryRollback.png)
 
-- Grafana login info
-  - admin
-  - change-me
+## Cleanup
 
-![Canary Dashboard](../../images/envoyCanaryDashboard.png)
+- Reset IMDb app.yaml to initial state
+
+  ```bash
+
+  cd $PIB_BASE
+  git checkout origin/main -- apps/imdb/app.yaml
+
+  git add .
+  git commit -am "reset imdb app.yaml"
+  git push
+
+  ```
+- Once you're finished with the workshop and experimenting, [follow these setps](../../outer-loop.md#delete-your-cluster) to delete your cluster
diff --git a/labs/images/envoyCanaryPromotion.png b/labs/images/envoyCanaryPromotion.png
diff --git a/labs/images/envoyCanaryRollback.png b/labs/images/envoyCanaryRollback.png
diff --git a/labs/outer-loop-aks-azure.md b/labs/outer-loop-aks-azure.md
@@ -43,7 +43,7 @@ git branch --show-current
   ```bash
 
   # get az user name and validate your role assignment
-  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/[@]+/_/g')
+  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/@.*//')
   az role assignment list --query "[].{principalName:principalName, roleDefinitionName:roleDefinitionName, scope:scope} | [? contains(principalName,'$principal_name')]" -o table
 
   ```

diff --git a/labs/outer-loop-arc-gitops.md b/labs/outer-loop-arc-gitops.md
@@ -52,7 +52,7 @@ git branch --show-current
   ```bash
 
   # get az user name and validate your role assignment
-  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/[@]+/_/g')
+  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/@.*//')
   az role assignment list --query "[].{principalName:principalName, roleDefinitionName:roleDefinitionName, scope:scope} | [? contains(principalName,'$principal_name')]" -o table
 
   ```

diff --git a/labs/outer-loop-multi-cluster.md b/labs/outer-loop-multi-cluster.md
@@ -43,7 +43,7 @@ git branch --show-current
   ```bash
 
   # get az user name and validate your role assignment
-  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/[@]+/_/g')
+  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/@.*//')
   az role assignment list --query "[].{principalName:principalName, roleDefinitionName:roleDefinitionName, scope:scope} | [? contains(principalName,'$principal_name')]" -o table
 
   ```

diff --git a/labs/outer-loop.md b/labs/outer-loop.md
@@ -81,7 +81,7 @@ git branch --show-current
   ```bash
 
   # get az user name and validate your role assignment
-  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/[@]+/_/g')
+  principal_name=$(az account show --query "user.name" --output tsv | sed -r 's/@.*//')
   az role assignment list --query "[].{principalName:principalName, roleDefinitionName:roleDefinitionName, scope:scope} | [? contains(principalName,'$principal_name')]" -o table
 
   ```

diff --git a/templates/pib-service-canary/canary.yaml b/templates/pib-service-canary/canary.yaml
@@ -27,7 +27,7 @@ spec:
       retryOn: "5xx"
   analysis:
     # schedule interval (default 60s)
-    interval: 20s
+    interval: 15s
     # max number of failed metric checks before rollback
     threshold: 5
     # max traffic percentage routed to canary

diff --git a/templates/pib-service-canary/ingressHttpProxy.yaml b/templates/pib-service-canary/ingressHttpProxy.yaml
@@ -9,3 +9,21 @@ spec:
   includes:
     - name: {{gitops.app}}
       namespace: {{gitops.namespace}}
+      conditions:
+      - prefix: /
+    - name: flagger-grafana
+      namespace: projectcontour
+      conditions:
+      - prefix: /grafana
+---
+
+apiVersion: projectcontour.io/v1
+kind: HTTPProxy
+metadata:
+  name: flagger-grafana
+  namespace: projectcontour
+spec:
+  routes:
+    - services:
+        - name: flagger-grafana
+          port: 8080