What file contains the GPU layers?

[JTMarsh556]

What file contains the gpu_layers setting? GPU utilization seems low and I want to see what this is set at. I went to the settings files and didnt see it there. I have seen older discussions that mention a privateGPT.py file but I cannot seem to locate that.

[thebrothbobo]

Hey mate, had the same issue. Found it in llm_component.py
privateGPT\private_gpt\components\llm\llm_component.py

You got a setting kwargs dictionnary :
settings_kwargs = {
"tfs_z": settings.llamacpp.tfs_z, # ollama and llama-cpp
"top_k": settings.llamacpp.top_k, # ollama and llama-cpp
"top_p": settings.llamacpp.top_p, # ollama and llama-cpp
"repeat_penalty": settings.llamacpp.repeat_penalty, # ollama llama-cpp
"n_gpu_layers": -1,
"offload_kqv": True
}

I tried few changes but still haven't succeed using GPU, maybe you'll have better luck

[JTMarsh556]

Thank you. I will let you know how it turns out.