Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opt_touch_all argument of dak.map_partitions should no longer be necessary #915

Closed
douglasdavis opened this issue Oct 24, 2023 · 0 comments · Fixed by #929
Closed

opt_touch_all argument of dak.map_partitions should no longer be necessary #915

douglasdavis opened this issue Oct 24, 2023 · 0 comments · Fixed by #929
Labels
bug Something isn't working

Comments

@douglasdavis
Copy link

douglasdavis commented Oct 24, 2023

The opt_touch_all argument to map_partitions was added to make layers in dask-awkward graphs manually touchable (that is, tell the optimization code to not rely on the typetracer graph for touching in that layer, just touch all awkward data at that point). This allowed us to workaround operations where automatic typetracer touching wasn't working. I think we're in a place now where we don't need it. I see it's used in coffea here:

opt_touch_all=False,


More info just for completeness:

The only place where we needed this was in dask_awkward.zip, where if we did something like:

upstream = get_data()
new_data = dak.zip({"foo": upstream.foo, "bar": upstream.bar})
new_data.foo.compute()

That compute would fail if we didn't also touch upstream's bar data, but bar wouldn't get touched by the compute on only foo, hence the need to manually touch the whole zipped up array.

I've made draft PR dask-contrib/dask-awkward#395 for testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant