Spark On K8S 0.5.1

What's Changed

• fix(webserver): fix the websocket URL by @hussein-awala in #36
• security: bump aiohttp to avoid CVE-2024-27306 by @hussein-awala in #37
• security: bump apache-airflow to avoid CVE-2024-31869 by @hussein-awala in #38
• security: bump gunicorn to avoid CVE-2024-1135 by @hussein-awala in #39

Full Changelog: 0.5.0...0.5.1

Spark On K8S 0.5.0

What's Changed

• feat(docker): add a dockerfile for spark-history server by @hussein-awala in #28
• feat(webserver): integrate spark history server by @hussein-awala in #29
• feat(helm): add support for spark history deployment in the helm chart by @hussein-awala in #30
• security: bump Apache Airflow version to avoid CVE-2024-29735 by @hussein-awala in #31
• chore(helm): add SPARK_ON_K8S_IN_CLUSTER by default to the webserver pod by @hussein-awala in #32
• ci(helm): publish the chart to a github page by @hussein-awala in #33
• doc: add instructions to install helm chart from new repo by @hussein-awala in #34
• security: update poetry constraints by @hussein-awala in #35

Full Changelog: 0.4.0...0.5.0

Spark On K8S 0.4.0

What's Changed

This release introduces new features for the API and webserver, including the ability to kill and delete Spark jobs and read driver pod logs directly from the webserver, and some improvements for the helm chart access service account permissions.

• feat(webserver): implement a UI to stream driver pod logs by @hussein-awala in #14
• chore: separate html templates and css styles by @hussein-awala in #15
• chore: separate sync and async app manager in two modules by @hussein-awala in #17
• feat: api routes to kill and delete spark app by @hussein-awala in #16
• fix: use websocket client in async kill_app to run pod exec by @hussein-awala in #18
• fix: remove duplicated app from app endpoints paths by @hussein-awala in #19
• feat: add kill and delete features to the webserver by @hussein-awala in #20
• fix: close the ws client properly by @hussein-awala in #21
• security: bump airflow version to 2.8.3 to avoid CVE-2024-28746 by @hussein-awala in #23
• feat(helm): support specific namespaces and swtiching between read and edit permissions by @hussein-awala in #24
• fix(webserver): fix the static folder path by @hussein-awala in #25
• fix(api): add missing websockets packages to api extra by @hussein-awala in #26
• chore(webserver): add a page to redirect to when there is a K8S error by @hussein-awala in #27

Full Changelog: 0.3.0...0.4.0

Spark On K8S 0.3.0

What's Changed

• feat: support driver and executor node selectors by @hussein-awala in #10
• feat: support driver and executor labels and annotations by @hussein-awala in #11
• feat: support driver pod tolerations by @hussein-awala in #12
• feat: support executor pod template path by @hussein-awala in #13

Full Changelog: 0.2.3...0.3.0

Spark On K8S 0.2.3

What's Changed

• fix(cli): skip adding api commands when api extra is not installed by @hussein-awala in #8
• fix(security): update flask-appbuilder to fix CVE-2024-25128 by @hussein-awala in #9

Full Changelog: 0.2.2...0.2.3

Spark On K8S 0.2.2

What's Changed

• doc: add missing parameters to docstring by @hussein-awala in #4
• fix(cli): Bump min Kubernetes version and fix setup namespace command by @hussein-awala in #3
• fix(cli): ENV_VARS_FROM_SECRET default value by @hussein-awala in #5
• fix(core): handle the case where secret_values is an empty dict by @hussein-awala in #6
• fix(cli): split the spark configs only by the first = by @hussein-awala in #7

Full Changelog: 0.2.1...0.2.2

Spark On K8S 0.2.1

What's Changed

• fix(security): bump cryptography version to fix CVE-2024-26130 by @hussein-awala in #2

Full Changelog: 0.2.0...0.2.1

Spark On K8S 0.2.0

What's Changed

• Support creating an ephemeral secret for sensitive arguments (Python client, CLI, and Airflow operator)
• Support mounting existing secrets to the driver pod as environment variables (Python client and CLI)
• Support mounting volumes to the spark driver pod (Python client and Airflow operator)
• Support mounting volumes to the spark executor pods (Python client and Airflow operator)

Security

• Add a lower bound for aiohttp to avoid CVE-2024-23829 and CVE-2024-23334
• Bump FastAPI and starlette versions to fix CVE-2024-24762

Full Changelog: 0.1.0...0.2.0