Change priority for scheduling reroute during timeout #16445
Conversation
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
|
❌ Gradle check result for 5e83a92: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
| "reroute after existing shards allocator timed out", | ||
| Priority.HIGH, | ||
| "reroute after existing shards allocator [R] timed out", | ||
| Priority.NORMAL, |
There was a problem hiding this comment.
Should we have a separate priority for primary vs replica?
There was a problem hiding this comment.
NORMAL also seems right for PSA. But during genuine issues in the cluster which can be identified with appropriate monitoring, we might need to raise it to HIGH. I will update the PR with a similar setting for ESA similar to BSA to raise reroute priority. Wdyt?
Updated |
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
imRishN
requested review from
anasalkouz,
andrross,
ashking94,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
jainankitk,
kotwanikunal,
linuxpi,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami and
VachaShah
as code owners
|
❌ Gradle check result for 6a448d0: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
imRishN
changed the title
|
❌ Gradle check result for 825a983: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
|
❌ Gradle check result for 5368e7f: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Rishab Nahata <rnnahata@amazon.com>
|
❌ Gradle check result for 2ba604d: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Description
This PR updates the priority of scheduling reroute when timed out from HIGH to NORMAL. This is because consistent HIGH reroutes might starve NORMAL priority tasks. And moreover, NORMAL is right for reasonable clusters. For clusters in messed up state which is causing NORMAL priority tasks to starve, we add a new dynamic cluster setting to raise the priority of reroute task to allocate shards in such scenarios.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
[ ] Functionality includes testing.[ ] API changes companion pull request created, if applicable.[ ] Public documentation issue/PR created, if applicable.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.