LoggingContext.tag being lost on metrics and traces

[matrixbot]

This issue has been migrated from #15920.

---

As discovered during the discussion in matrix-org/synapse#15708 (comment),

Background

LoggingContext.tag was introduced to differentiate what kind of sync was happening in our metrics. The only place we populate it is for this use case even though it could be used generally:

https://github.com/matrix-org/synapse/blob/a4243183f0b500f9f30f2d24af19f30a99f65f63/synapse/handlers/sync.py#L348-L357

We also use it in the tracing with SynapseTags.REQUEST_TAG which says that it's "used to distinguish full vs incremental syncs, etc".

Problem

If I look at the Prometheus data, I don't see any data for matrix.org. I only see it for one other instance we track which is running Synapse 1.83.0. Has it broken since then?

And if I look at the traces, request_tag is always set to the default empty string when I look at those SyncRestServlet traces (also checked RoomInitialSyncRestServlet).

It seems like the context is being lost somewhere.