chore(pipeline) : use Sentry as main notifier #369
Conversation
vperron
force-pushed
the
vperron/notify-sentry
branch
from
9738fc3 to
728b6ae
Compare
There was a problem hiding this comment.
presque envie de simplement utiliser https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/errors.html
pipeline/dags/dag_utils/sentry.py
Outdated
| with configure_scope() as scope: | ||
| dag_id = context.get("dag").dag_id | ||
| task_id = context.get("task_instance").task_id | ||
| execution_date = context.get("execution_date") |
There was a problem hiding this comment.
execution_date c'est déprécié depuis bien longtemps
pipeline/dags/dag_utils/sentry.py
Outdated
| from sentry_sdk import configure_scope | ||
|
|
||
|
|
||
| def task_failure_callback(context): |
There was a problem hiding this comment.
c'est assez hacky, puisque la callback ne fait rien d'autres que configurer une scope
There was a problem hiding this comment.
oui, le nommage est naze, j'ai poussé hier avant de partir mais j'aurais dû faire un draft, toutes mes excuses 🙏
| @@ -229,6 +229,7 @@ resource "null_resource" "up" { | |||
| AIRFLOW_WWW_USER_PASSWORD='${var.airflow_admin_password}' | |||
| AIRFLOW__CORE__FERNET_KEY='${var.airflow__core__fernet_key}' | |||
| AIRFLOW__SENTRY__SENTRY_DSN='${var.airflow__sentry__sentry_dsn}' | |||
| AIRFLOW__SENTRY__RELEASE='${var.stack_version}' | |||
There was a problem hiding this comment.
c'est parce que la variable release de la conf de sentry permet de lier une version du soft en question:
https://docs.sentry.io/product/sentry-basics/integrate-backend/configuration-options/
C'est peut etre un peu overkill, on peut juste mettre un tag avec le SHA et une URL vers le commit, mais je me disais que utiliser des maintenant la notion de "release" est un plus (les issues sont automatiquement liées au commit suspect, etc)
|
Si on utilise simplement le logging de base (comme aujourd'hui donc) on perd a minima le lien direct vers les logs Airflow depuis l'issue, que l'on avait jusqu'à aujourd'hui dans Mattermost. Maintenant là où tu as raison c'est que on peut probablement utiliser le Je trouve que configurer une release apporte un net plus car on peut immédiatement dire quel commit a fait apparaitre ou réapparaitre tel probleme. Enfin, tagger le nom du DAG et de la tâche est négligeable mais permet d'identifier super rapidement dans le bandeau de |
Sentry is a much better tool than our little code executor, at least in terms of not spamming our alerting channel. We can precisely configure how and when we want to be notified, what is important, discuss about issues, etc. It also now directly links every issue to a particular commit, making it easier to identify regressions. The flow changes a little bit as the logs are in the Sentry issue, not directly linked in the Slack message, but I hope we'll be able to triage much better.
vperron
force-pushed
the
vperron/notify-sentry
branch
from
728b6ae to
ab05c24
Compare
Sentry is a much better tool than our little code executor, at least in terms of not spamming our alerting channel. We can precisely configure how and when we want to be notified, what is important, discuss about issues, etc.
It also now directly links every issue to a particular commit, making it easier to identify regressions.
The flow changes a little bit as the logs are in the Sentry issue, not directly linked in the Slack message, but I hope we'll be able to triage much better.
The goal here is to use the opportunity of migrating to Slack to kill 2 birds with one stone.
Example of the custom info added to Sentry :