Skip to content

Installing Python Packages in the Dockerfile

Jump to bottom
Sean Sy edited this page Oct 12, 2021 · 1 revision

Python Package Installation Directory

Note that when installing Python packages on the Dockerfile

FROM apache/airflow:1.10.15-python3.8
USER root
RUN pip install --no-cache-dir --user --no-warn-script-location -r requirements.txt

the pip install command may install the packages in different locations on the container depending on the flags specified. For example, when running pip install as root, packages are installed in /usr/local/lib/pythonX.Y/site-packages. But when running pip install --user as root, the packages will be installed in /root/.local/lib/pythonX.Y/site-packages. This will make the packages inaccessible to the airflow user.

While installing without the --user flag makes some packages available to all Linux user accounts, some packages such as the Airflow backport providers packages (e.g. apache-airflow-backport-providers-google) require for the package to be installed in the same directory as the Airflow installation ($AIRFLOW_USER_HOME_DIR/.local/lib/pythonX.Y/site-packages/airflow/). To fix any such issues, be sure to have the installed packages available to the airflow user by having these package files available in $AIRFLOW_USER_HOME_DIR/.local/lib/pythonX.Y/site-packages/.

Clone this wiki locally