Would it be possible to have an option to set the maximum VRAM usage instead of trying to guess number of layers? #257
TiagoTiago
started this conversation in
Ideas
Replies: 1 comment 1 reply
-
|
We don't know the vram in advance since it depends on the model and further settings. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
What if you start with just a small number of layers, and then extrapolate from that? Or do layers have wildly different sizes inside a model? Do you need to know the number of layers before you start sending them to the GPU, or could you check how much VRAM is used after each layer is added, and stop when the next layer would cross the threshold, switching to the CPU from that point onwards? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
The trial and error process is a bit annoying...
Beta Was this translation helpful? Give feedback.
All reactions