Support for Airflow version 1.X

[conchaox]

Hello,

We've been using this chart for more than two years now with Airflow version 2.x, I've a recent requirement for Airflow version 1.X and I haven't been able to make it work. All the pods are coming up except for the web on, here's the log:

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
/home/airflow/.local/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:813: UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
  'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 76, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1076, in webserver
    app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 300, in cached_app
    app, _ = create_app(config, session, testing)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 109, in create_app
    update_perms=conf.getboolean('webserver', 'UPDATE_FAB_PERMS'))
  File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/base.py", line 148, in __init__
    self.init_app(app, session)
  File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/base.py", line 202, in init_app
    self.sm = self.security_manager_class(self)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/www_rbac/security.py", line 179, in __init__
    super(AirflowSecurityManager, self).__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/security/sqla/manager.py", line 51, in __init__
    super(SecurityManager, self).__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/security/manager.py", line 249, in __init__
    from flask_oauthlib.client import OAuth
ModuleNotFoundError: No module named 'flask_oauthlib'

Right now, I'm using chart version 8.7.0 but it's supposed that most of the chart versions support version 1? We are using Terraform to deploy the module and for our most recent module versions we are overriding the web config in this way:

webserverConfig:
   stringOverride: |-
     from flask_appbuilder.security.manager import AUTH_OAUTH
     import os

     AUTH_TYPE = AUTH_OAUTH

     AUTH_USER_REGISTRATION = True

     AUTH_USER_REGISTRATION_ROLE = "Public"

     OAUTH_PROVIDERS = [
         {
             "name": "okta",
             "icon": "fa-circle-o",
             "token_key": "access_token",
             "remote_app": {
                 "client_id": os.getenv("OKTA_CLIENT_ID"),
                 "client_secret": os.getenv("OKTA_CLIENT_SECRET"),
                 "api_base_url": "https://company.okta.com/oauth2/v1/",
                 "server_metadata_url": "https://company.okta.com/.well-known/openid-configuration",
                 "client_kwargs": {
                     "scope": "openid profile email groups"
                 },
                 "access_token_url": "https://company.okta.com/oauth2/v1/token",
                 "authorize_url": "https://company.okta.com/oauth2/v1/authorize",
             }
         }
     ]

     AUTH_ROLES_MAPPING = {
%{for okta_group, airflow_role in var.group_permissions~}
         "${okta_group}": ["${airflow_role}"],
%{endfor~}
     }

     # replace ALL the user's roles each login
     AUTH_ROLES_SYNC_AT_LOGIN = ${var.roles_sync}

     # force users to re-auth after 30min of inactivity
     PERMANENT_SESSION_LIFETIME = 1800

I noticed that for old chart versions (e.g. 7.16.0) that have Airflow version 1.X as default value webserverConfig doesn't even exists, so, we cannot get rid of this web config since we use it to authenticate using Okta, so how can we make this work?

Thanks!

[thesuperzapper]

@conchaox I think you might have missed a few of the warnings on the Integrate Airflow with LDAP or OAUTH page.

1. To use AUTH_ROLES_MAPPING you need to use Flask-Appbuilder>=3.2.0
   • You should be able to upgrade your Flask-Appbuilder version with Airflow 1.10.12+ using pip install
   • I am not sure what the latest version that will work is, probably start with v3.4.5 (which is the latest version <4)
2. To use OAUTH, you will need to install the Authlib pip package (similar to above)

There is one thing you will need to add to your webserverConfig.stringOverride which is only needed when using Airflow 1.10.X:

# only needed for airflow 1.10
from airflow import configuration as conf
SQLALCHEMY_DATABASE_URI = conf.get("core", "SQL_ALCHEMY_CONN")

[conchaox]

Hi @thesuperzapper that worked! thanks for pointing it out, however, I'm not sure why the login window still asking for user and password, I though setting the airflow.users value to [] would do the trick.

My config looks like this now:

airflow:
  legacyCommands: True
  image:
    repository: apache/airflow
    tag: 1.10.12-python3.6
  users: []
web:
  extraPipPackages:
    - "Flask-AppBuilder~=3.3.0"
    - "Authlib~=0.15.3"
  webserverConfig:
    stringOverride: |-
      from airflow import configuration as conf
      from flask_appbuilder.security.manager import AUTH_OAUTH
      import os

      SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')

[thesuperzapper]

@conchaox setting users: [] just means there are no AUTH_DB users, which is probably what you want when using AUTH_OAUTH (so people can only log in with OAUTH).

However, removing users from airflow.users will NOT delete their corresponding user in airflow itself, so remember to disable/delete them in the airflow admin UI.

[conchaox]

oh, it seems that all the webServerConfig override deletes the Okta part and just kept these:

from airflow import configuration as conf
from flask_appbuilder.security.manager import AUTH_OAUTH
import os

SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')