CUSOLVER (dense): cache workspace in fat handle

[bjarthur]

riffing off of #2279 for getrf, getrs, sytrf, sytrs and friends. much cleaner API than #2464.

[bjarthur]

tests pass locally but i have not battle tested this yet

[bjarthur]

ready for review. tests really pass locally now, and it works well in my application.

[maleadt]

Superficially LGTM, but I don't have the time for a thorough review right now.

This probably should also integrate with the reclaim_hooks in order to wipe the caches when running out of memory (both here and in the other fat handles).
EDIT: let's move this to a separate issue.

[amontoison]

@bjarthur I really like the idea of a "fat" handle!
We discussed about it with @maleadt regarding CUSPARSE, aiming to avoid allocating a new buffer for each sparse matrix product, but we never got the chance to implement it.

@maleadt, I’m wondering if CUDA.jl consistently reuses the same handle.
I know we can have up to 32 handles, but for efficiency, we should reuse the one that stored the buffer from the previous factorization.
Otherwise, we'll end up storing a lot of unnecessary workspaces.

If that’s not the case, wouldn’t it be better to implement a cache of workspaces where the "fat" handle could pick a workspace from the pool based on a specific "id" related to the routine?

[maleadt]

I’m wondering if CUDA.jl consistently reuses the same handle.
I know we can have up to 32 handles, but for efficiency, we should reuse the one that stored the buffer from the previous factorization.
Otherwise, we'll end up storing a lot of unnecessary workspaces.

We do as long as you're using the same task, which I assume you are. In that case, calling handle() will always return the same object, and only a single handle will be cached.

[maleadt]

Forgot about this; rebased to give it another CI run.