[PL-133119] Topological sort before rewriting aggregate jobs #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Given the leaf jobs `packages.foo` & `packages.bar`, an aggregate job
`aggregate0` with
_hydraGlobConstituents = true;
constituents = [ "packages.*" ];
and an aggregate job `aggregate1` with
constituents = [ "aggregate0" ];
then it may happen depending on the order of evaluation that `aggregate1`
depends on the old derivation of `aggregate0` (i.e. the one without
rewritten constituents) and doesn't depend on `packages.foo` and
`packages.bar` because it was rewritten before `aggregate0` was
rewritten.
Using a toposort (derived from the one in CppNix's libutil) to solve
that. Cycles are a hard error, then _all_ aggregates are failed to make
sure we don't finish one of those too early.
This also extracts the code into a few helper functions and a slightly
better data structure, otherwise this would've gotten too chaotic.
ctheune
approved these changes
Nov 14, 2024
ctheune
reviewed
Nov 20, 2024
| for (auto job = jobs.begin(); job != jobs.end(); job++) { | ||
| // If all jobs are selected by an aggregate job, select all | ||
| // jobs except itself. | ||
| if (childJobName == "*" && job.key() == jobName) { |
There was a problem hiding this comment.
I think we can drop the "*" comparison and just always filter out the parent job.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Given the leaf jobs
packages.foo&packages.bar, an aggregate jobaggregate0withand an aggregate job
aggregate1withthen it may happen depending on the order of evaluation that
aggregate1depends on the old derivation ofaggregate0(i.e. the one without rewritten constituents) and doesn't depend onpackages.fooandpackages.barbecause it was rewritten beforeaggregate0was rewritten.Using a toposort (derived from the one in CppNix's libutil) to solve that. Cycles are a hard error, then all aggregates are failed to make sure we don't finish one of those too early.
This also extracts the code into a few helper functions and a slightly better data structure, otherwise this would've gotten too chaotic.