chore(gpu): rework select to avoid using local streams

[agnesLeroy]

closes: please link all relevant issues

PR content/description

Check-list:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• Relevant issues are marked as resolved/closed, related issues are linked in the description
• Check for breaking changes (including serialization changes) and add them to commit message following the conventional commit specification

[agnesLeroy]

Latency on H100 is slightly improved. I need to check the effect on the whitepaper ERC20 transfer throughput.

[guillermo-oyarzun]

Does this improves the issues in the multi-gpu?

[agnesLeroy]

Nope, it doesn't 😞

[agnesLeroy]

I think the multi-GPU throughput issue is related to the use of cudaDeviceSynchronize in drop for CudaVec, I haven't tried to check though. Disabling that synchronization may lead to memory errors... I could try nevertheless to confirm it has an effect 🤔

[guillermo-oyarzun]

I think the multi-GPU throughput issue is related to the use of cudaDeviceSynchronize in drop for CudaVec, I haven't tried to check though. Disabling that synchronization may lead to memory errors... I could try nevertheless to confirm it has an effect 🤔

Confirmation would be nice, even though changing that safely might involve a big refactor.

[agnesLeroy]

Hmm looks like it's not that: https://github.com/zama-ai/tfhe-rs/actions/runs/12349715463/job/34461608896. I disabled cudaDeviceSynchronize in drop there but still we see the same effect. We need to investigate further.