Skip to content

Commit

Permalink
docs: update template documentation
Browse files Browse the repository at this point in the history
Signed-off-by: Simon Brugman <sfbbrugman@gmail.com>
  • Loading branch information
sbrugman committed Jun 16, 2023
1 parent 2b785da commit 23036e1
Showing 1 changed file with 31 additions and 18 deletions.
49 changes: 31 additions & 18 deletions kedro-airflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,30 +56,43 @@ You can use the additional command line argument `--jinja-file` (alias `-j`) to
kedro airflow create --jinja-file=./custom/template.j2
```

Similarly, the included DAG `kwargs` template can be custom provided:

```bash
kedro airflow create --kwargs-template=./custom/kwargs_template.j2
```

#### How can I pass arguments to the Airflow DAGs dynamically?

`kedro-airflow` picks up configuration from `airflow.yml` files. Arguments can be specified globally, or per pipeline:

```yaml
# Example of global arguments for Airflow DAGs
kwargs: |
start_date=datetime(2023, 1, 1),
max_active_runs=3,
# Global parameters
default:
start_date: [2023, 1, 1]
max_active_runs: 3
# https://airflow.apache.org/docs/stable/scheduler.html#dag-runs
schedule_interval="@once",
catchup=False,
schedule_interval: "@once"
catchup: false
# Default settings applied to all tasks
default_args=dict(
owner="airflow",
depends_on_past=False,
email_on_failure=False,
email_on_retry=False,
retries=1,
retry_delay=timedelta(minutes=5)
),
# Example configuration per pipeline (overrides `kwargs` from above)
default_args:
owner: "airflow"
depends_on_past: false
email_on_failure: false
email_on_retry: false
retries: 1
retry_delay: 5

# Arguments specific to the pipeline (overrides the parameters above)
data_science:
kwargs: |
schedule_interval="@once",
default_args:
owner: "airflow-ds"
```
Arguments can also be passed via `--params` in the command line:

```bash
kedro airflow create --params "schedule_interval='@weekly'"
```

These variables are passed to the Jinja2 template.

0 comments on commit 23036e1

Please sign in to comment.