Inspire Matomo: visits per day DAG #40
Conversation
| parameters = get_parameters( | ||
| period=period, date=date, endpoint_key="visits_per_day" | ||
| ) | ||
| token = os.environ.get("MATOMO_AUTH_TOKEN") |
There was a problem hiding this comment.
It's better to use the airflow variables for this kind of things.
There was a problem hiding this comment.
left the comment regarding this below👇
README.md
Outdated
| @@ -70,6 +70,7 @@ airflow standalone | |||
| ### 6. Start Postgres with Docker Compose | |||
|
|
|||
| If you're using Docker to manage your Postgres database, start the service. | |||
| IMPORTANT: Please add MATOMO_AUTH_TOKEN value in Docker compose file | |||
There was a problem hiding this comment.
why do we need that in Docker compose file? If we use airlfow we can just have a fixture as we do in https://github.com/inspirehep/backoffice/blob/main/workflows/scripts/variables/variables.json and the we can replace it on the airflow interface
There was a problem hiding this comment.
Since the beginning of the project, we've been using environment variables to maintain consistency. I think that moving to Airflow variables was discussed a long time ago, but it was left as a "to improve later" feature 🤔
I agree that using Airflow variables is a great idea, just it will require a different approach to secrets management on Kubernetes. Airflow variables are stored in the Airflow metadata database, which means we would need to:
- Remove secrets from Kubernetes YAML files.
- Export them via the UI (Admin > Variables), OR set them directly from the command line, OR use a JSON file as done in the back office.
If we decide to implement this change, we should create a new task to migrate all secrets and variables to be managed through Airflow variables
ErnestaP
force-pushed
the
inspire-matomo
branch
3 times, most recently
from
1582d78
to
6d808f2
Compare
ErnestaP
force-pushed
the
inspire-matomo
branch
from
6d808f2
to
d5f29df
Compare
No description provided.