Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Continuos replication #2896

Merged
merged 26 commits into from
Oct 15, 2024
Merged

Continuos replication #2896

merged 26 commits into from
Oct 15, 2024

Conversation

manbearpig1996
Copy link
Contributor

@manbearpig1996 manbearpig1996 commented Sep 18, 2024 •
edited
Loading

This PR contains -

  1. Changing the logic of replication cycle. Currently in a replication cycle, we will create groups and then wait for all
    groups to finish. With this change, we all replica groups will run multiple iterations until any group reaches a predefined limit.
  2. Added logic to terminate cycle, when we are adding/removing replicas or shutting the thread down.
  3. Added tests for group generation, active replication & leader based replication.
  4. If we set limit to 1, then the new logic will work exactly as same as old logic.
  5. New metric for tracking throttled replica count.
  6. logs and converting trackers to string for printing.
  7. Integration of storeKeyConverter drop keys logic to replication logic.
  8. Refactoring of old replicate method to remove redundant statements.
  9. Refactoring of logic of checking replica status to single place.

Navneet Sachan added 16 commits September 5, 2024 11:43
adding logic in replicate method
ee0244b
Merge branch 'master' into continuous-replication-replicate
557117e
Adding logic for group generation and added comments and tests
69a94aa
modifying comments
daa03c7
termination logic for continuous replication
8e3d070
changing language of comments to present tense
85a9888
Merge branch 'continuous-replication-replicate' into continuous-repli…
56f9929 
…cation-termination
Adding comments for methods
e2a431c
Merge branch 'master' into continuous-replication-termination-2
51bd386
Adding comments for group tracker
70ead19
adding logic for group creation for continuous replication
809c0f8
adding test for group generation
f0be250
adding logic to recheck state of ACTIVE replicas too after finishing
b3c502e
add dummy comment
eb25871
adding logic for continuous replication
9a5b353
adding comments and logs to code
bc293bf
@manbearpig1996 manbearpig1996 marked this pull request as ready for review September 19, 2024 13:56
@manbearpig1996 manbearpig1996 changed the title Continuos replicate Continuos replication Sep 19, 2024
mudit-saxena
mudit-saxena reviewed Sep 30, 2024
View reviewed changes
ambry-api/src/main/java/com/github/ambry/config/ReplicationConfig.java Outdated Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/continuous/ActiveGroupTracker.java Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/continuous/DataNodeTracker.java Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/continuous/DataNodeTracker.java Outdated Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/continuous/DataNodeTracker.java Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/continuous/GroupTracker.java Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/ReplicaThread.java Outdated Show resolved Hide resolved
ambry-replication/src/main/java/com/github/ambry/replication/ReplicaThread.java Show resolved Hide resolved
mudit-saxena
mudit-saxena approved these changes Oct 7, 2024
View reviewed changes
Copy link

@mudit-saxena mudit-saxena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GTG

vaibhav-mittal-linkedin
vaibhav-mittal-linkedin approved these changes Oct 8, 2024
View reviewed changes
Copy link
Contributor

@vaibhav-mittal-linkedin vaibhav-mittal-linkedin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

aga9900
aga9900 approved these changes Oct 10, 2024
View reviewed changes
Copy link

@aga9900 aga9900 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

piyujai
piyujai approved these changes Oct 10, 2024
View reviewed changes
Copy link
Contributor

@piyujai piyujai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@manbearpig1996 manbearpig1996 merged commit 08552e8 into master Oct 15, 2024
5 checks passed
@manbearpig1996 manbearpig1996 deleted the continuos-replicate branch October 15, 2024 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vaibhav-mittal-linkedin vaibhav-mittal-linkedin vaibhav-mittal-linkedin approved these changes

@mudit-saxena mudit-saxena mudit-saxena approved these changes

@piyujai piyujai piyujai approved these changes

@aga9900 aga9900 aga9900 approved these changes

@justinlin-linkedin justinlin-linkedin Awaiting requested review from justinlin-linkedin

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@manbearpig1996 @aga9900 @piyujai @mudit-saxena @vaibhav-mittal-linkedin