Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory

[royalsrivastavagithub]

Hey guys ,

i figuring out how node-llama-cpp works
and i got this error :

ggml_vulkan: Found 1 Vulkan devices:
Vulkan0: NVIDIA GeForce RTX 3050 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32
ggml_vulkan: Device memory allocation of size 4413526016 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
[node-llama-cpp] ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 4413526016

how to fix this ?

[giladgd]

What's your code?
Try to scaffold a project from a template and run it: https://node-llama-cpp.withcat.ai/guide/#scaffold-new-project

[royalsrivastavagithub]

another thing i just downloaded f16 version of llama3.2 1b from suggested repo of huggingface https://huggingface.co/mradermacher an it worked without a problem no errors

[giladgd]

The log you mentioned is a warning log generated by llama.cpp.
If the model works as expected then you can safely ignore it, but if you encounter any issues with it then this error indicates that you may need to find and switch to a newer model conversion that's compatible with the latest llama.cpp changes.

[royalsrivastavagithub]

ok now i get it so i have to get llama.cpp and the fp16 model and there is a convert script which i can use to convert the model into compatible one ?

[giladgd]

You better use a pre-converted model instead of converting one yourself since creating a high-quality conversion/quantization is not straightforward, and it can be very time-consuming to get it right.
I don't recommend using an f16 model since it's unnecessarily large for doing inference and will be significantly slower.
Using a quantized version such as Q4_K_M or Q8_0 for the highest quality is much better than using f16.
You can see a table with notes for each file on each model card for more information, like this one for example.

[royalsrivastavagithub]

got it ! . i think that error is not causing any problems thanks for your support .