denoise3d can't allocate memory for cpu run

[wangrsc]

Hello Everyone, I tried to run denoise3d with (-d -1) to use cpu only. The MRC file is 10221440500 voxels. I got the following error message:
RuntimeError: [enforce fail at alloc_cpu.cpp:117] err == 0. DefaultCPUAllocator: can't allocate memory: you tried to allocate 87205216256 bytes. Error code 12 (Cannot allocate memory)
But free -h showed the following:
total used free shared buff/cache available
Mem: 250Gi 136Gi 110Gi 69Mi 4.7Gi 112Gi
Swap: 31Gi 282Mi 31Gi

free memory is 110G, larger than 87,205,216,256 bytes.
Any suggestions to make it run?

Thanks.

Liguo

[tbepler]

I'm not sure exactly when you measured the free RAM relative to when the error occurred, but the error just reports how much RAM torch was trying to allocate in the call that ran out of RAM. It's likely that there were already other tensors in RAM taking up enough space that the 87 gigs couldn't be allocated. It's also possible that if you have other software running at the same time, less RAM was free when topaz was running.

[wangrsc]

After rebooting, it runs well on CPU only. I tried to run it on Nvidia Geoforce 4090 with 24G memory for 1022x1440x500, but always run into CUDA outof memory error. Is there any way to run denoise3D on gpu?

[tbepler]

Have you looked at the --patch-size option? I suggest setting it to something like 256.

[wangrsc]

patch-size 256 didn't work. But patch-size 128 worked. Does the patch-size affect the result?

[tbepler]

Only if --patch-padding is too small. Do you see any stitching artifacts in the denoised tomogram?

[wangrsc]

I compared the output with that using patch-size 512 on CPU, and found the contrast in patch-size 128 image using GPU is not as good as that in patch-size 512 image using CPU with the same padding 48.