The performance of my model becomes much worse when using ACA backpropagation in DataParallel wrapper.

[wadx2019]

Hello, I am a graduate student. I tried to execute my project based on torch_ACA solver in multiple gpus with DP wrapper recently. However, I found that the performance will decrease much compared with that in a single gpu, while the naive backpropagation still works. Can you give me some instructions or possible reasons?

[juntang-zhuang]

I believe it's because there's no proper error tolerance or grad reduce operation in the case of data parallel if you have a distributed setup. Sorry I did not wrote that since I don't have much machine.