-
-
Notifications
You must be signed in to change notification settings - Fork 14k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python3Packages.llama-cpp-python: init at 0.2.18 #268712
Conversation
I just found this PR when I was about to create one. Because they differ in some aspects, I still want to mention it here to facilitate discussion. This PR uses the existing llama-cpp package which has the advantage that cuda, rocm etc. should work. The (current) drawback is that it does not use the vendored llama.cpp version and requires patches. |
There is an issue with the shared library change in llama-cpp causing problems when using it with ollama, I'll investigate that issue. |
Thanks for starting this PR @elohmeier ! Currently this is missing a few dependencies to get https://github.com/abetlen/llama-cpp-python#openai-compatible-web-server to run: uvicorn, fastapi, starlette, and pydantic 2.x ( #244564 ). |
@elohmeier are you still seeing an issue here? llama-cpp-python has moved on a bit, and the patches no longer apply. |
Unfortunately I have no use for that package right now. Maybe someone else can pick this up. |
i have a working version for latest llama-cpp-python, with CUDA support, but using vendored llama, if people are interested can create PR |
It's been almost a year since this PR was open and Would it now make sense to add this library to @kirillrdy , can you share this version you mentioned ? |
Description of changes
Added the llama-cpp-python package containing the Python bindings for llama.cpp. Required shared libraries and models have been exposed in llama-cpp.
Fixes #242792.
Things done
nix.conf
? (See Nix manual)sandbox = relaxed
sandbox = true
nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)