Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(core/df-repr): better upper bound pruning to reduce plan space #265

Merged
merged 7 commits into from
Dec 19, 2024
Merged

feat(core/df-repr): better upper bound pruning to reduce plan space #265

merged 7 commits into from
Dec 19, 2024

Conversation

skyzh
Copy link
Member

@skyzh skyzh commented Dec 14, 2024 •
edited
Loading

  • Remove cross join from the repr
  • Implement upper bound pruning correctly
  • Two-stage cascades, also use the new task graph instead of refactoring the current one

skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(df-repr): remove cross join, use inner join (#265)
5aa9209 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from b3288bc to 7fef116 Compare December 18, 2024 02:32
@skyzh skyzh marked this pull request as ready for review December 18, 2024 02:32
@skyzh skyzh requested review from jurplel and yliang412 December 18, 2024 02:33
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(core): prune based on upper bound (#265)
ef3beae 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>

tune plan space budget

Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
test: reproduce hash join planning issue (#265)
254b2e8 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from 7fef116 to b3cff3c Compare December 18, 2024 02:34
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(core): rewrite tasks using generators, better pruning (#265)
583adab 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(df-repr): adjust filter cost to prefer hash join (#265)
b3cff3c 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh mentioned this pull request Dec 18, 2024
Closed
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(df-repr): adjust filter cost to prefer hash join (#265)
fc187cf 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from b3cff3c to fc187cf Compare December 18, 2024 02:35
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
fix(core): reduce group id when updating winner (#265)
cac5d3f 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from f225728 to cac5d3f Compare December 18, 2024 02:39
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
fix(core): do not apply logical rule if iter budget is exausted (#265)
bf25604 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
fix(core): do not apply logical rule if iter budget is exausted (#265)
7f7c84d 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from bf25604 to 7f7c84d Compare December 18, 2024 04:24
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
fix(core): do not apply logical rule if iter budget is exausted (#265)
5325531 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from 7f7c84d to 5325531 Compare December 18, 2024 04:25
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(df-repr): remove cross join, use inner join (#265)
648b41a 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(core): prune based on upper bound (#265)
fa6817c 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>

tune plan space budget

Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
test: reproduce hash join planning issue (#265)
9dfc530 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(core): rewrite tasks using generators, better pruning (#265)
e67776b 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from 5325531 to e5f7c50 Compare December 18, 2024 04:45
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
refactor(df-repr): adjust filter cost to prefer hash join (#265)
483981c 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
fix(core): reduce group id when updating winner (#265)
db0fca6 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 18, 2024
@skyzh
fix(core): do not apply logical rule if iter budget is exausted (#265)
e5f7c50 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
yliang412
yliang412 reviewed Dec 18, 2024
View reviewed changes
Copy link
Member

@yliang412 yliang412 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed:

  • cross join to inner join changes
  • main upper bound pruning logic in optimize_inputs_inner

I need another look at task2.rs and potentially doc more aggressively like #234. I actually like this version of the task graph a lot (no state machine logic).

Is there any other part of the logic I need pay special attention to?

@skyzh
Copy link
Member Author

skyzh commented Dec 18, 2024

I think part of this PR is tuning to ensure the plan is what we want (i.e., filter selectivity and behavior on reaching the budget). The tasks2.rs logic is also something that we should fully understand as we are going to bring more logic into it (phys prop, etc.)

yliang412
yliang412 approved these changes Dec 19, 2024
View reviewed changes
Copy link
Member

@yliang412 yliang412 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to doc the task graph a little bit more but LGTM

@skyzh skyzh enabled auto-merge (rebase) December 19, 2024 02:04
skyzh added 7 commits December 18, 2024 21:04
@skyzh
refactor(df-repr): remove cross join, use inner join (#265)
9b120fd 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh
refactor(core): prune based on upper bound (#265)
5bb101a 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>

tune plan space budget

Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh
test: reproduce hash join planning issue (#265)
4b90d28 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh
refactor(core): rewrite tasks using generators, better pruning (#265)
60cb62e 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh
refactor(df-repr): adjust filter cost to prefer hash join (#265)
8741544 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh
fix(core): reduce group id when updating winner (#265)
900c386 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh
fix(core): do not apply logical rule if iter budget is exausted (#265)
023d47e 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/plan-join-perf branch from e5f7c50 to 023d47e Compare December 19, 2024 02:04
@skyzh skyzh merged commit d77b3e3 into main Dec 19, 2024
1 check passed
@skyzh skyzh deleted the skyzh/plan-join-perf branch December 19, 2024 02:13
skyzh added a commit that referenced this pull request Dec 19, 2024
@skyzh
refactor(df-repr): remove cross join, use inner join (#265)
e599545 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 19, 2024
@skyzh
refactor(core): prune based on upper bound (#265)
b32964b 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>

tune plan space budget

Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 19, 2024
@skyzh
test: reproduce hash join planning issue (#265)
52a4000 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 19, 2024
@skyzh
refactor(core): rewrite tasks using generators, better pruning (#265)
5531542 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 19, 2024
@skyzh
refactor(df-repr): adjust filter cost to prefer hash join (#265)
4407728 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
skyzh added a commit that referenced this pull request Dec 19, 2024
@skyzh
fix(core): reduce group id when updating winner (#265)
6497fe7 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yliang412 yliang412 yliang412 approved these changes

@jurplel jurplel Awaiting requested review from jurplel

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@skyzh @yliang412