Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

LoggingContext.tag being lost on metrics and traces #15920

Open
MadLittleMods opened this issue Jul 11, 2023 · 0 comments
Open

LoggingContext.tag being lost on metrics and traces #15920

MadLittleMods opened this issue Jul 11, 2023 · 0 comments
Labels
A-Logging Synapse's logs (structured or otherwise). Not metrics. A-Metrics metrics, measures, stuff we put in Prometheus O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact

Comments

@MadLittleMods
Copy link
Contributor

As discovered during the discussion in #15708 (comment),

Background

LoggingContext.tag was introduced to differentiate what kind of sync was happening in our metrics. The only place we populate it is for this use case even though it could be used generally:

if since_token is None:
sync_type = "initial_sync"
elif full_state:
sync_type = "full_state_sync"
else:
sync_type = "incremental_sync"
context = current_context()
if context:
context.tag = sync_type

We also use it in the tracing with SynapseTags.REQUEST_TAG which says that it's "used to distinguish full vs incremental syncs, etc".

Problem

If I look at the Prometheus data, I don't see any data for matrix.org. I only see it for one other instance we track which is running Synapse 1.83.0. Has it broken since then?

And if I look at the traces, request_tag is always set to the default empty string when I look at those SyncRestServlet traces (also checked RoomInitialSyncRestServlet).

It seems like the context is being lost somewhere.

@MadLittleMods MadLittleMods added A-Logging Synapse's logs (structured or otherwise). Not metrics. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. A-Metrics metrics, measures, stuff we put in Prometheus labels Jul 11, 2023
@reivilibre reivilibre added S-Minor Blocks non-critical functionality, workarounds exist. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact O-Uncommon Most users are unlikely to come across this or unexpected workflow labels Jul 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-Logging Synapse's logs (structured or otherwise). Not metrics. A-Metrics metrics, measures, stuff we put in Prometheus O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. Z-Dev-Wishlist Makes developers' lives better, but doesn't have direct user impact
Projects
None yet
Development

No branches or pull requests

2 participants