Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update full-auto migration tool #2529

Merged

Conversation

Arun-LinkedIn
Copy link
Contributor

  1. Add semi-auto placement as preference lists in Resource config in Helix before Full-Auto migration. This causes helix to not rebalance immediately when Full-Auto is turned on and gives us control to migrate few partitions at a time.
  2. Add support to remove few (or all) partitions at a time from preference lists in Resource config. Sample command to use the tool is /export/apps/jdk/JDK-1_8_0_121/bin/java -Dlog4j2.configurationFile=file:./log4j2.xml -cp ambry.jar com.github.ambry.clustermap.HelixBootstrapUpgradeTool --clusterName Ambry-EI --dcs ei-ltx1 --zkLayoutPath zkLayoutEI.json --resourceName 10001 --partitions 180,183,189,196,201 --updatePreferenceLists

@codecov-commenter
Copy link

codecov-commenter commented Aug 2, 2023

Codecov Report

Attention: 205 lines in your changes are missing coverage. Please review.

Comparison is base (fdc4ec2) 32.40% compared to head (a15ac8a) 19.52%.
Report is 2 commits behind head on master.

Files Patch % Lines
...ub/ambry/clustermap/HelixBootstrapUpgradeUtil.java 11.29% 157 Missing ⚠️
...ub/ambry/clustermap/HelixBootstrapUpgradeTool.java 0.00% 48 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2529       +/-   ##
=============================================
- Coverage     32.40%   19.52%   -12.89%     
+ Complexity     4460     2787     -1673     
=============================================
  Files           806      806               
  Lines         65491    65599      +108     
  Branches       7994     8015       +21     
=============================================
- Hits          21225    12809     -8416     
- Misses        42031    51609     +9578     
+ Partials       2235     1181     -1054     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@justinlin-linkedin justinlin-linkedin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, can you please add another command to switch FULLAUTO back to SEMI AUTO.

Comment on lines 1223 to 1225
// 3. Set semi-auto placement as preference list so that no replicas are moved.
Map<String, List<String>> semiAutoPreferenceLists = idealState.getPreferenceLists();
resourceConfig.setPreferenceLists(semiAutoPreferenceLists);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have another parameter to control this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@Arun-LinkedIn
Copy link
Contributor Author

BTW, can you please add another command to switch FULLAUTO back to SEMI AUTO.

Sure. Added command to switch back to SEMI AUTO

Arun Bhima added 2 commits November 28, 2023 17:47
2.Add option to populate semi-auto placement as preference lists during Full-Auto migration
3.Add option to roll back to semi auto

4.Add changes to add configs only without enabling Full Auto

5.Add changes for migrating video cluster
@Arun-LinkedIn Arun-LinkedIn merged commit 7257e95 into linkedin:master Jan 29, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants