There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird.

@dxenes1 I wonder if you encountered the same on the download side?

There appear to be some nasties w.r.t numpy shared memory when slicing and uploading a large array. I think the default behavior might be to copy the array across each thread, which is no good for RAM. Was really hoping to avoid doing a deeper dive here but looks like we might need to... I'm also seeing a bunch of 500 errors in the test logs which is weird.

@dxenes1 I wonder if you encountered the same on the download side?

Didn't see anything with download, maybe since I'm downloading raw bytearrays and then constructing the numpy array at the end i don't run into the same issue when downloading.