Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parallelism to BossRemote#create_cutout #64

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

j6k4m8
Copy link
Member

@j6k4m8 j6k4m8 commented Jul 27, 2020

By default, uses n_cpu parallel threads to upload. Now symmetric with get_cutout which has been parallel since v1.0.0.

Fixes #62

intern/service/boss/v1/volume.py Outdated Show resolved Hide resolved
@j6k4m8
Copy link
Member Author

j6k4m8 commented Mar 21, 2021

There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird.

@dxenes1 I wonder if you encountered the same on the download side?

movestill
movestill previously approved these changes Mar 21, 2021
@dxenes1
Copy link
Contributor

dxenes1 commented Mar 22, 2021

There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird.

@dxenes1 I wonder if you encountered the same on the download side?

Didn't see anything with download, maybe since I'm downloading raw bytearrays and then constructing the numpy array at the end i don't run into the same issue when downloading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelize BossRemote#create_cutout
3 participants