Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: [Rhythm] Full RF1 read path #4478

Draft
wants to merge 3 commits into
base: main-rhythm
Choose a base branch
from

Conversation

mdisibio
Copy link
Contributor

What this PR does:
I believe these are the changes needed to migrate the backend read path to RF1. New config options rf1_after which is a timestamp, after that only blocks with rf==1 are included in searches and trace by ID lookups. And an option to discontinue flushes to object storage from the ingesters.

How each api is handled:

Search: the frontend determines the blocks:

query_frontend:
  search:
    rf1_after: "2024-12-18T00:00:00Z"

Trace lookup: the querier determines the blocks:

querier:
  trace_by_id:
    rf1_after: "2024-12-18T00:00:00Z"

Tags: this is actually not filtering on replication factor, so no changes needed
Metrics: already limited to rf1 blocks

The migration path:

  • Enable kafka ingest, block-builder, etc. It is now flushing RF1 blocks.
  • Grace period for monitoring and validation
  • Set rf1_after to move the read path over to the new RF1 blocks (recent searches/tags/tracelookups are still directed at ingesters)
  • Grace period for monitoring and validation
  • Disable flushes on ingesters

Note: Global config is non-trivial due to Tempo's modular setup, so this adds redundant config in multiple places. Not ideal.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Member

@mapno mapno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, so close to RF1 🚀

userID: instance.instanceID,
blockID: op.blockID,
}, false)
if i.cfg.FlushObjectStorage {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to mark blocks flushed or change some logic so complete blocks are cleared at some point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants