Skip to content

Latest commit

 

History

History
124 lines (85 loc) · 5.92 KB

testing_packages.rst

File metadata and controls

124 lines (85 loc) · 5.92 KB

Manually building and testing release candidate packages

Breeze can be used to test new release candidates of packages - both Airflow and providers. You can easily configure the CI image of Breeze to install and start Airflow for both Airflow and provider packages, whether they are built from sources or downloaded from PyPI as release candidates.

The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).

Prerequisites

The way to test it is rather straightforward:

  1. Make sure that the packages - both airflow and providers are placed in the dist folder of your Airflow source tree. You can either build them there or download from PyPI (see the next chapter).
  2. You can run breeze shell or breeze start-airflow commands with adding the following flags - --mount-sources remove, --use-packages-from-dist, and --use-airflow-version wheel/sdist. The first one removes the airflow source tree from the container when starting it, the second one installs airflow and providers packages from the dist folder when entering breeze, and the third one specifies the package's format (either wheel or sdist). Omitting the latter will result in skipping the installation of the package(s), and a consequent error when later importing them.

Testing pre-release packages

There are two ways how you can get Airflow packages in dist folder - by building them from sources or downloading them from PyPI.

Note

Make sure you run rm dist/* before you start building packages or downloading them from PyPI because the packages built there already are not removed manually.

In order to build apache-airflow from sources, you need to run the following command:

breeze release-management prepare-airflow-package

In order to build providers from sources, you need to run the following command:

breeze release-management prepare-provider-packages <PROVIDER_1> <PROVIDER_2> ... <PROVIDER_N>

The packages are built in dist folder and the command will summarise what packages are available in the dist folder after it finishes.

If you want to download the packages from PyPI, you need to run the following command:

pip download apache-airflow-providers-<PROVIDER_NAME>==X.Y.Zrc1 --dest dist --no-deps

You can use it for both release and pre-release packages.

Examples of testing pre-release packages

Few examples below explain how you can test pre-release packages, and combine them with locally build and released packages.

The following example downloads apache-airflow and celery and kubernetes provider packages from PyPI and eventually starts Airflow with the Celery Executor. It also loads example dags and default connections:

rm dist/*
pip download apache-airflow==2.9.0rc1 --dest dist --no-deps
pip download apache-airflow-providers-celery==3.6.2rc1 --dest dist --no-deps
pip download apache-airflow-providers-cncf-kubernetes==8.1.0rc1 --dest dist --no-deps
breeze start-airflow --mount-sources remove --use-packages-from-dist --use-airflow-version sdist --executor CeleryExecutor --backend postgres --load-default-connections --load-example-dags

The following example downloads celery and kubernetes provider packages from PyPI, builds apache-airflow package from the main sources and eventually starts Airflow with the Celery Executor. It also loads example dags and default connections:

rm dist/*
breeze release-management prepare-airflow-package
pip download apache-airflow-providers-celery==3.6.2rc1 --dest dist --no-deps
pip download apache-airflow-providers-cncf-kubernetes==8.1.0rc1 --dest dist --no-deps
breeze start-airflow --mount-sources remove --use-packages-from-dist --use-airflow-version sdist --executor CeleryExecutor --backend postgres --load-default-connections --load-example-dags

The following example builds celery, kubernetes provider packages from the main sources, downloads 2.9.0 version of apache-airflow package from PyPI and eventually starts Airflow using default executor for the backend chosen (no example dags, no default connections):

rm dist/*
breeze release-management prepare-provider-packages celery cncf.kubernetes
pip download apache-airflow==2.9.0 --dest dist --no-deps
breeze start-airflow --mount-sources remove --use-packages-from-dist --use-airflow-version sdist

You can mix and match packages from PyPI (final or pre-release candidates) with locally build packages. You can also choose which providers to install this way since the --mount-sources remove flag makes sure that Airflow installed does not contain all the providers - only those that you explicitly downloaded or built in the dist folder. This way you can test all the combinations of Airflow and Providers you might need.

For other kinds of tests look at Testing document