chore: release 8.9.0 (#859)

* docs: update README Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> * feat: update default container tags Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> * docs: update kubernetes-api links to `1.29` Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> * feat: update default airflow tag to `2.8.4-python3.9` Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> * chore: release 8.9.0 Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> * docs: update CHANGELOG (typo) Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> --------- Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com>
airflow-helm · May 1, 2024 · ed99218 · ed99218
1 parent d4624fa
commit ed99218
Show file tree

Hide file tree

Showing 16 changed files with 216 additions and 186 deletions.
diff --git a/charts/airflow/CHANGELOG.md b/charts/airflow/CHANGELOG.md
@@ -8,19 +8,48 @@ This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 TBD
 
-## [8.8.0] - 2023-08-28
+## [8.9.0] - 2024-04-30
 
 > 🟨 __NOTES__ 🟨
 >
-> - the default airflow image is now `apache/airflow:2.6.3-python3.9`, but you can still use any supported version of airflow, see the [airflow version support matrix](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support)
-> - this is the first version of the chart with support for airflow 2.7, along with pretty much all previous versions of airflow, see the [airflow version support matrix](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support)
+> - this is the first version of the chart with support for airflow 2.9, along with [pretty much all previous versions of airflow](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support)
+> - the default airflow image is now `apache/airflow:2.8.4-python3.9`, but you can still use any supported version of airflow
+> - we have fixed an important bug for airflow <=2.5.3 (introduced in chart version 8.7.1), some liveness probes were creating significant load on the backend database [#853](https://github.com/airflow-helm/charts/pull/853)
 
 > 🟦 __OTHER__ 🟦
 >
 > - If you appreciate the `User-Community Airflow Helm Chart` please consider supporting us!
 >    - [give a ⭐ on GitHub](https://github.com/airflow-helm/charts/stargazers)
 >    - [give a ⭐ on ArtifactHub](https://artifacthub.io/packages/helm/airflow-helm/airflow)
 
+### Changed
+- the default airflow image is now `apache/airflow:2.8.4-python3.9` (see the [airflow version support matrix](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support))
+- the default git-sync image is now `registry.k8s.io/git-sync/git-sync:v3.6.9`
+- the default pgbouncer image is now `ghcr.io/airflow-helm/pgbouncer:1.22.1-patch.0`
+- the default embedded postgres image is now `ghcr.io/airflow-helm/postgresql-bitnami:11.22-patch.0`
+- the default embedded redis image is now `bitnami/redis:6.2.14-debian-12-r17`
+
+### Added
+- add `extraContainers` and `extraInitContainers` values for all components ([#856](https://github.com/airflow-helm/charts/pull/856))
+- add `topologySpreadConstraints` values  ([#752](https://github.com/airflow-helm/charts/pull/752))
+- add values to set init-container resources ([#855](https://github.com/airflow-helm/charts/pull/855))
+- add `airflow.defaultContainerSecurityContext` value ([#624](https://github.com/airflow-helm/charts/pull/624))
+- add `pgbouncer.statsUsers` value to set `stats_user` in pgbouncer ([#820](https://github.com/airflow-helm/charts/pull/820))
+- add optional `rbac.secrets` value to give GET/LIST/WATCH on Secrets ([#790](https://github.com/airflow-helm/charts/pull/790))
+
+### Fixed
+- scheduler and triggerer probe performance for airflow <2.6.0 ([#853](https://github.com/airflow-helm/charts/pull/853))
+- fix `extraPipPackages` values for airflow 2.9.0+ ([#858](https://github.com/airflow-helm/charts/pull/858))
+- don't import `User` and `Role` FAB models, fixes airflow 2.9.0+ ([#857](https://github.com/airflow-helm/charts/pull/857))
+- set `appProtocol` on Service ports (for Istio) ([#854](https://github.com/airflow-helm/charts/pull/854))
+
+## [8.8.0] - 2023-08-28
+
+> 🟨 __NOTES__ 🟨
+>
+> - this is the first version of the chart with support for airflow 2.7, along with [pretty much all previous versions of airflow](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support)
+> - the default airflow image is now `apache/airflow:2.6.3-python3.9`, but you can still use any supported version of airflow
+
 ### Changed
 - the default airflow image is now `apache/airflow:2.6.3-python3.9` (see the [airflow version support matrix](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support))
 
@@ -29,7 +58,6 @@ TBD
 - support `include_deferred` in pool sync for airflow 2.7.0 ([#775](https://github.com/airflow-helm/charts/pull/775))
 
 ### Fixed
-
 - sync-users for airflow 2.7.0 ([#772](https://github.com/airflow-helm/charts/pull/772))
 - invalid pgbouncer-certs volume spec when empty ([#777](https://github.com/airflow-helm/charts/pull/777))
 
@@ -40,7 +68,6 @@ TBD
 > - this is the first version of the chart with support for airflow 2.6, along with pretty much all previous versions of airflow, see the [airflow version support matrix](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support)
 
 ### Fixed
-
 - fixed liveness probes in airflow 2.6.0 ([#743](https://github.com/airflow-helm/charts/pull/743))
 
 ## [8.7.0] - 2023-04-06
@@ -784,7 +811,8 @@ TBD
 >
 > - To read about versions `7.0.0` and before, please see the [legacy repo](https://github.com/helm/charts/tree/master/stable/airflow).
 
-[Unreleased]: https://github.com/airflow-helm/charts/compare/airflow-8.8.0...HEAD
+[Unreleased]: https://github.com/airflow-helm/charts/compare/airflow-8.9.0...HEAD
+[8.9.0]: https://github.com/airflow-helm/charts/compare/airflow-8.8.0...airflow-8.9.0
 [8.8.0]: https://github.com/airflow-helm/charts/compare/airflow-8.7.1...airflow-8.8.0
 [8.7.1]: https://github.com/airflow-helm/charts/compare/airflow-8.7.0...airflow-8.7.1
 [8.7.0]: https://github.com/airflow-helm/charts/compare/airflow-8.6.1...airflow-8.7.0

diff --git a/charts/airflow/Chart.yaml b/charts/airflow/Chart.yaml
@@ -1,8 +1,8 @@
 apiVersion: v2
 description: Airflow Helm Chart (User Community) - the standard way to deploy Apache Airflow on Kubernetes with Helm
 name: airflow
-version: 8.8.0
-appVersion: 2.6.3
+version: 8.9.0
+appVersion: 2.8.4
 icon: https://avatars.githubusercontent.com/u/71061241
 home: https://github.com/airflow-helm/charts/tree/main/charts/airflow
 maintainers:

diff --git a/charts/airflow/README.md b/charts/airflow/README.md
@@ -46,18 +46,6 @@ Originally created in 2017, it has since helped thousands of companies create pr
 
 <br>
 
-## History
-
-The `User-Community Airflow Helm Chart` chart has a long history of being the standard way to deploy Apache Airflow on Kubernetes.
-
-Here is a brief overview of the chart's development from 2017 until today:
-
-- From October 2017 until December 2018, the chart was called `kube-airflow` and was developed in [`gsemet/kube-airflow`](https://github.com/gsemet/kube-airflow)
-- From December 2018 until November 2020, the chart was called `stable/airflow` and was developed in [`helm/charts`](https://github.com/helm/charts/tree/master/stable/airflow)
-- Since November 2020, the chart has been called `Airflow Helm Chart (User Community)` and is developed in `airflow-helm/charts`
-
-Please note, this chart is __independent__ from the official chart in the `apache/airflow` repo, which was forked from Astronomer's proprietary chart in May 2021.
-
 ## Project Goals
 
 1. Ease of Use
@@ -68,7 +56,7 @@ Please note, this chart is __independent__ from the official chart in the `apach
 ## Key Features
 
 - __Support for Airflow Versions:__ 
-   - [`1.10` | `2.0` | `2.1` | `2.2` | `2.3` | `2.4` | `2.5` | `2.6` | `2.7`](#airflow-version-support)
+   - [`1.10` | `2.0` | `2.1` | `2.2` | `2.3` | `2.4` | `2.5` | `2.6` | `2.7` | `2.8` | `2.9`](#airflow-version-support)
 - __Support for Airflow Executors:__ 
    - [`CeleryExecutor` | `KubernetesExecutor` | `CeleryKubernetesExecutor`](#airflow-executor-support)
 - __Easily Connect with your Database:__
@@ -95,6 +83,18 @@ Please note, this chart is __independent__ from the official chart in the `apach
    - [`Heartbeat Check`](https://github.com/airflow-helm/charts/tree/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md#scheduler-heartbeat-check) |
      [`Task Creation Check`](https://github.com/airflow-helm/charts/tree/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md#scheduler-task-creation-check)
 
+## History
+
+The `User-Community Airflow Helm Chart` chart has a long history of being the standard way to deploy Apache Airflow on Kubernetes.
+
+Here is a brief overview of the chart's development from 2017 until today:
+
+- From October 2017 until December 2018, the chart was called `kube-airflow` and was developed in [`gsemet/kube-airflow`](https://github.com/gsemet/kube-airflow)
+- From December 2018 until November 2020, the chart was called `stable/airflow` and was developed in [`helm/charts`](https://github.com/helm/charts/tree/master/stable/airflow)
+- Since November 2020, the chart has been called `Airflow Helm Chart (User Community)` and is developed in `airflow-helm/charts`
+
+> Please note, this chart is __independent__ from the official chart in the `apache/airflow` repo, which was forked from Astronomer's proprietary chart in May 2021.
+
 <br>
 
 ## Guides
@@ -158,17 +158,19 @@ Please note, this chart is __independent__ from the official chart in the `apach
 
 The following table lists the __airflow versions__ supported by this chart (set the version with [`airflow.image.tag`](https://github.com/airflow-helm/charts/tree/main/charts/airflow/docs/faq/configuration/airflow-version.md) value).
 
-Chart Version → <br> Airflow Version ↓  | `7.0.0` - `7.16.0` | `8.0.0` - `8.5.3` | `8.6.0` | `8.6.1 - 8.7.0` | `8.7.1` | `8.8.0+`
---- | --- | --- | --- | --- | --- | ---
-`1.10.X` | ✔️ | ✔️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub>
-`2.0.X` | ❌ | ✔️ | ✔️ | ✔️ | ✔️️ | ✔️️
-`2.1.X` | ❌ | ✔️ | ✔️ | ✔️ | ✔️️ | ✔️️
-`2.2.X` | ❌ | ⚠️ <sub>[2]</sub> | ✔️️ | ✔️ | ✔️️ | ✔️️
-`2.3.X` | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️
-`2.4.X` | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️
-`2.5.X` | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️
-`2.6.X` | ❌ | ❌ | ❌ | ❌ | ✔️️ | ✔️️
-`2.7.X` | ❌ | ❌ | ❌ | ❌ | ❌ | ✔️️
+Chart Version → <br> Airflow Version ↓  | `7.0.0` - `7.16.0` | `8.0.0` - `8.5.3` | `8.6.0` | `8.6.1` - `8.7.0` | `8.7.1` | `8.8.0` | `8.9.0+`
+--- | --- | --- | --- | --- | --- | --- | ---
+`1.10.X` | ✔️ | ✔️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub> | ✔️️ <sub>[1]</sub>
+`2.0.X` | ❌ | ✔️ | ✔️ | ✔️ | ✔️️ | ✔️️ | ✔️️
+`2.1.X` | ❌ | ✔️ | ✔️ | ✔️ | ✔️️ | ✔️️ | ✔️️
+`2.2.X` | ❌ | ⚠️ <sub>[2]</sub> | ✔️️ | ✔️ | ✔️️ | ✔️️ | ✔️️
+`2.3.X` | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️ | ✔️️
+`2.4.X` | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️ | ✔️️
+`2.5.X` | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️ | ✔️️
+`2.6.X` | ❌ | ❌ | ❌ | ❌ | ✔️️ | ✔️️ | ✔️️
+`2.7.X` | ❌ | ❌ | ❌ | ❌ | ❌ | ✔️️ | ✔️️
+`2.8.X` | ❌ | ❌ | ❌ | ❌ | ❌ | ✔️️ | ✔️️
+`2.9.X` | ❌ | ❌ | ❌ | ❌ | ❌ | ❌️ | ✔️️
 
 <sub>[1] you must set `airflow.legacyCommands = true` when using airflow version `1.10.X`</sub><br>
 <sub>[2] the [Deferrable Operators & Triggers](https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html) feature won't work, as there is no `airflow triggerer` Deployment</sub>

diff --git a/charts/airflow/docs/faq/configuration/airflow-plugins.md b/charts/airflow/docs/faq/configuration/airflow-plugins.md
@@ -23,10 +23,10 @@ You may embed your [Airflow Plugins](https://airflow.apache.org/docs/apache-airf
 
 This chart uses the official [`apache/airflow`](https://hub.docker.com/r/apache/airflow) Docker images.
 
-Here is a Dockerfile that extends `apache/airflow:2.6.3-python3.9` with custom plugins:
+Here is a Dockerfile that extends `apache/airflow:2.8.4-python3.9` with custom plugins:
 
 ```dockerfile
-FROM apache/airflow:2.6.3-python3.9
+FROM apache/airflow:2.8.4-python3.9
 
 # plugin files can be copied under `/home/airflow/plugins`
 # (where `./plugins` is relative to the docker build context)

diff --git a/charts/airflow/docs/faq/configuration/extra-python-packages.md b/charts/airflow/docs/faq/configuration/extra-python-packages.md
@@ -192,10 +192,10 @@ You may embed your python packages directly into the container image.
 
 This chart uses the official [`apache/airflow`](https://hub.docker.com/r/apache/airflow) Docker images.
 
-Here is a Dockerfile that extends `apache/airflow:2.6.3-python3.9` with the `torch` package:
+Here is a Dockerfile that extends `apache/airflow:2.8.4-python3.9` with the `torch` package:
 
 ```dockerfile
-FROM apache/airflow:2.6.3-python3.9
+FROM apache/airflow:2.8.4-python3.9
 
 # install your pip packages
 RUN pip install --no-cache-dir \

diff --git a/charts/airflow/docs/faq/dags/load-dag-definitions.md b/charts/airflow/docs/faq/dags/load-dag-definitions.md
@@ -249,10 +249,10 @@ You may embed your DAG files directly into the container image.
 
 This chart uses the official [`apache/airflow`](https://hub.docker.com/r/apache/airflow) Docker images.
 
-Here is a Dockerfile that extends `apache/airflow:2.6.3-python3.9` by placing DAG files into `/opt/airflow/dags`:
+Here is a Dockerfile that extends `apache/airflow:2.8.4-python3.9` by placing DAG files into `/opt/airflow/dags`:
 
 ```dockerfile
-FROM apache/airflow:2.6.3-python3.9
+FROM apache/airflow:2.8.4-python3.9
 
 ## copy the content of local folder `./my_dag_folder` into container folder `/opt/airflow/dags`
 COPY ./my_dag_folder /opt/airflow/dags

diff --git a/charts/airflow/docs/faq/kubernetes/affinity-node-selectors-tolerations.md b/charts/airflow/docs/faq/kubernetes/affinity-node-selectors-tolerations.md
@@ -21,7 +21,7 @@ airflow:
     # my_node_label_1: value1
     # my_node_label_2: value2
 
-  ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core
+  ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#affinity-v1-core
   defaultAffinity: {}
     # podAffinity:
     #   requiredDuringSchedulingIgnoredDuringExecution:
@@ -44,7 +44,7 @@ airflow:
     #                 - S2
     #         topologyKey: topology.kubernetes.io/zone
 
-  ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#toleration-v1-core
+  ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#toleration-v1-core
   defaultTolerations: []
     # - key: "key1"
     #   operator: "Exists"

diff --git a/charts/airflow/docs/faq/kubernetes/mount-environment-variables.md b/charts/airflow/docs/faq/kubernetes/mount-environment-variables.md
@@ -4,7 +4,7 @@
 
 # Mount Environment Variables from Secrets/ConfigMaps
 
-You may use the `airflow.extraEnv` value to mount extra environment variables with the same structure as [EnvVar in ContainerSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#envvar-v1-core).
+You may use the `airflow.extraEnv` value to mount extra environment variables with the same structure as [EnvVar in ContainerSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#envvar-v1-core).
 
 > 🟦 __Tip__ 🟦
 >

diff --git a/charts/airflow/docs/faq/kubernetes/mount-files.md b/charts/airflow/docs/faq/kubernetes/mount-files.md
@@ -13,13 +13,13 @@ For example, to mount a Secret called `redshift-creds` at the `/opt/airflow/secr
 ```yaml
 airflow:
   extraVolumeMounts:
-    ## spec for VolumeMount: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core
+    ## spec for VolumeMount: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volumemount-v1-core
     - name: redshift-creds
       mountPath: /opt/airflow/secrets/redshift-creds
       readOnly: true
 
   extraVolumes:
-    ## spec for Volume: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core
+    ## spec for Volume: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volume-v1-core
     - name: redshift-creds
       secret:
         secretName: redshift-creds
@@ -56,13 +56,13 @@ For example, to mount a Secret called `redshift-creds` at the `/opt/airflow/secr
 ```yaml
 workers:
   extraVolumeMounts:
-    ## spec for VolumeMount: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core
+    ## spec for VolumeMount: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volumemount-v1-core
     - name: redshift-creds
       mountPath: /opt/airflow/secrets/redshift-creds
       readOnly: true
   
   extraVolumes:
-    ## spec for Volume: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core
+    ## spec for Volume: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volume-v1-core
     - name: redshift-creds
       secret:
         secretName: redshift-creds
@@ -77,13 +77,13 @@ For example, to mount a Secret called `redshift-creds` at the `/opt/airflow/secr
 ```yaml
 airflow:
   kubernetesPodTemplate:
-    ## spec for VolumeMount: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core
+    ## spec for VolumeMount: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volumemount-v1-core
     extraVolumeMounts:
       - name: redshift-creds
         mountPath: /opt/airflow/secrets/redshift-creds
         readOnly: true
   
-    ## spec for Volume: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core
+    ## spec for Volume: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volume-v1-core
     extraVolumes:
       - name: redshift-creds
         secret:

diff --git a/charts/airflow/docs/faq/monitoring/log-persistence.md b/charts/airflow/docs/faq/monitoring/log-persistence.md
@@ -241,12 +241,12 @@ For example, to mount a [`persistentVolumeClaim`](https://kubernetes.io/docs/con
 ```yaml
 airflow:
   extraVolumeMounts:
-    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core
+    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volumemount-v1-core
     - name: logs-volume
       mountPath: /opt/airflow/logs
 
   extraVolumes:
-    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core
+    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volume-v1-core
     - name: logs-volume
       persistentVolumeClaim:
         claimName: my-existing-persistent-volume-claim
@@ -281,12 +281,12 @@ For example, to mount an [`nfs`](https://kubernetes.io/docs/concepts/storage/vol
 ```yaml
 airflow:
   extraVolumeMounts:
-    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core
+    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volumemount-v1-core
     - name: logs-volume
       mountPath: /opt/airflow/logs
 
   extraVolumes:
-    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core
+    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volume-v1-core
     - name: logs-volume
       nfs:
         path: /path/on/nfs/server
@@ -326,12 +326,12 @@ For example, to mount a [`hostPath`](https://kubernetes.io/docs/concepts/storage
 ```yaml
 airflow:
   extraVolumeMounts:
-    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volumemount-v1-core
+    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volumemount-v1-core
     - name: logs-volume
       mountPath: /opt/airflow/logs
 
   extraVolumes:
-    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#volume-v1-core
+    ## spec: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.29/#volume-v1-core
     - name: logs-volume
       hostPath:
         ## WARNING: this represents a local path on the Kubernetes Node