-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[server] Add threadsafe mode to venice-server which adjusts message processing order #910
Open
ZacAttack
wants to merge
12
commits into
linkedin:main
Choose a base branch
from
ZacAttack:threadsafe
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Commits on Mar 21, 2024
-
[server][WIP] Add threadsafe mode to venice-server which adjusts mess…
…age processing order This is an initial phase PR. It is seen as the minimal set of changes needed in order to add a mode where writes on leader are committed to rocksdb prior to producing. This change in order has the following impacts: -Drainer is skipped on leaders: In a later refactor it might be prudent to remove the drainer entirely. However, in order to best accomodate that, it would likely make sense to execute a kind of batching logic when flushing to rocksdb. We do not attempt to make this change in this PR. -DCR logic must change Since writes are persisted to rocksdb prior to producing to Kafka, we now must accomodate for the possibility of left over state on a leader. To address this, we add a new mode to the merge conflict resolution logic where upon a perfect tie (on value and timestamp), we resolve to produce the repeated record. The intention here is to be able to be certain that a write which was persisted to rocksdb on leader but not produced doesn't end up getting lost due to failing DCR. -Transient Record is disabled transient record cache is disabled for those ingestion tasks which enable this mode. This is itself was one of the goals, but we should go here with some validation. Most clusters in production end up seeing pretty low cache hit rate on transient record cache in production, however, there is at least one use case that gets as high as a 20% hit rate. Theoretically, we may be able to avoid taking too much hit here as we are able to give the memory savings to rocksdb cache, but this needs vetting. If this doesn't work, then we will need to replace the transient record cache with a simple size/time based cache. There are also some cleanups here and there. Getting rid of some code paths that we no longer need and cleaning up others. NOTE: Integration tests haven't been completely added to this PR yet. Part of that is because while switching some of the existing integration tests to this mode, some tests are failing. This needs some more diagnosis. Hence the WIP tag.
Configuration menu - View commit details
-
Copy full SHA for 7b7a328 - Browse repository at this point
Copy the full SHA 7b7a328View commit details
Commits on Mar 26, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 7c8178c - Browse repository at this point
Copy the full SHA 7c8178cView commit details
Commits on Apr 10, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0d4e6a0 - Browse repository at this point
Copy the full SHA 0d4e6a0View commit details
Commits on Apr 11, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 88adca9 - Browse repository at this point
Copy the full SHA 88adca9View commit details
Commits on Apr 25, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 593cd0f - Browse repository at this point
Copy the full SHA 593cd0fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c0e60c - Browse repository at this point
Copy the full SHA 8c0e60cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3af747c - Browse repository at this point
Copy the full SHA 3af747cView commit details
Commits on Jun 20, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 75633d7 - Browse repository at this point
Copy the full SHA 75633d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2e517e - Browse repository at this point
Copy the full SHA d2e517eView commit details
Commits on Jul 17, 2024
-
Configuration menu - View commit details
-
Copy full SHA for fe74f0a - Browse repository at this point
Copy the full SHA fe74f0aView commit details
Commits on Jul 31, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 99451e1 - Browse repository at this point
Copy the full SHA 99451e1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ce894d - Browse repository at this point
Copy the full SHA 0ce894dView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.