Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server] Add threadsafe mode to venice-server which adjusts message processing order #910

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Commits on Mar 21, 2024

  1. [server][WIP] Add threadsafe mode to venice-server which adjusts mess…

    …age processing order
    
    This is an initial phase PR.  It is seen as the minimal set of changes needed in order to add a mode where writes on leader are committed to rocksdb prior to producing.  This change in order has the following impacts:
    
    -Drainer is skipped on leaders:
     In a later refactor it might be prudent to remove the drainer entirely.  However, in order to best accomodate that, it would likely make sense to execute a kind of batching logic when flushing to rocksdb.  We do not attempt to make this change in this PR.
    
    -DCR logic must change
     Since writes are persisted to rocksdb prior to producing to Kafka, we now must accomodate for the possibility of left over state on a leader.  To address this, we add a new mode to the merge conflict resolution logic where upon a perfect tie (on value and timestamp), we resolve to produce the repeated record.  The intention here is to be able to be certain that a write which was persisted to rocksdb on leader but not produced doesn't end up getting lost due to failing DCR.
    
    -Transient Record is disabled
     transient record cache is disabled for those ingestion tasks which enable this mode.  This is itself was one of the goals, but we should go here with some validation.  Most clusters in production end up seeing pretty low cache hit rate on transient record cache in production, however, there is at least one use case that gets as high as a 20% hit rate.  Theoretically, we may be able to avoid taking too much hit here as we are able to give the memory savings to rocksdb cache, but this needs vetting.  If this doesn't work, then we will need to replace the transient record cache with a simple size/time based cache.
    
    There are also some cleanups here and there.  Getting rid of some code paths that we no longer need and cleaning up others.
    
    NOTE: Integration tests haven't been completely added to this PR yet.  Part of that is because while switching some of the existing integration tests to this mode, some tests are failing.  This needs some more diagnosis.  Hence the WIP tag.
    ZacAttack committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    7b7a328 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. Configuration menu
    Copy the full SHA
    7c8178c View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Configuration menu
    Copy the full SHA
    0d4e6a0 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. address comments

    ZacAttack committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    88adca9 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Configuration menu
    Copy the full SHA
    593cd0f View commit details
    Browse the repository at this point in the history
  2. reduce timeouts

    ZacAttack committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    8c0e60c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3af747c View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. Configuration menu
    Copy the full SHA
    75633d7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d2e517e View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Configuration menu
    Copy the full SHA
    fe74f0a View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. Configuration menu
    Copy the full SHA
    99451e1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0ce894d View commit details
    Browse the repository at this point in the history