python3Packages.llama-cpp-python: init at 0.2.18

[elohmeier]

Description of changes

Added the llama-cpp-python package containing the Python bindings for llama.cpp. Required shared libraries and models have been exposed in llama-cpp.
Fixes #242792.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[7omb]

I just found this PR when I was about to create one. Because they differ in some aspects, I still want to mention it here to facilitate discussion.

This PR uses the existing llama-cpp package which has the advantage that cuda, rocm etc. should work. The (current) drawback is that it does not use the vendored llama.cpp version and requires patches.

[elohmeier]

There is an issue with the shared library change in llama-cpp causing problems when using it with ollama, I'll investigate that issue.

[mausch]

Thanks for starting this PR @elohmeier !

Currently this is missing a few dependencies to get https://github.com/abetlen/llama-cpp-python#openai-compatible-web-server to run: uvicorn, fastapi, starlette, and pydantic 2.x ( #244564 ).

[414owen]

@elohmeier are you still seeing an issue here? llama-cpp-python has moved on a bit, and the patches no longer apply.

[elohmeier]

Unfortunately I have no use for that package right now. Maybe someone else can pick this up.

[kirillrdy]

i have a working version for latest llama-cpp-python, with CUDA support, but using vendored llama, if people are interested can create PR

[hoh]

It's been almost a year since this PR was open and llama-cpp is now well in nixpkgs.

Would it now make sense to add this library to nixpkgs ?

@kirillrdy , can you share this version you mentioned ?

[kirillrdy]

@hoh #349657

[hoh]

@hoh #349657

Nice 🤩, thank you!
I'll test it on Monday.