Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: private map partitions #477

Merged
merged 3 commits into from
Feb 27, 2024
Merged

Conversation

lgray
Copy link
Collaborator

@lgray lgray commented Feb 22, 2024

This PR splits map_parititions into itself and an unsafe version _map_partitions which does not make the effort to sanitize input functions or data.

This avoids creating an ArgsKwargsPackedFunction when it is not necessary to do so in routine operations like getitem and fancy indexing.

Saves O(1)s in a complex analysis.

@martindurant
Copy link
Collaborator

I see nothing wrong with this at all. Where do we document (for ourselves) when we can call this internal method?

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 96.42857% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 93.10%. Comparing base (8cb8994) to head (dd319b6).
Report is 16 commits behind head on main.

Files Patch % Lines
src/dask_awkward/lib/core.py 96.42% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #477      +/-   ##
==========================================
+ Coverage   93.06%   93.10%   +0.03%     
==========================================
  Files          23       23              
  Lines        3290     3305      +15     
==========================================
+ Hits         3062     3077      +15     
  Misses        228      228              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lgray
Copy link
Collaborator Author

lgray commented Feb 22, 2024

@martindurant added a helpful description.

@lgray
Copy link
Collaborator Author

lgray commented Feb 26, 2024

anything blocking this one?

@martindurant
Copy link
Collaborator

Let's push this one in, and I'll deal with the caching PR

@lgray lgray merged commit 3ad9b49 into dask-contrib:main Feb 27, 2024
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants