Add llama.cpp backend

[baptistecolle]

Add llama.cpp as Backend for Optimum Benchmark

Overview

This PR introduces llama.cpp as a backend for the Optimum benchmark (see issue #117).

Changes

• Added an example in the examples folder demonstrating how to run the llama.cpp backend:

defaults:
  - benchmark
  - scenario: inference
  - launcher: inline
  - backend: llama_cpp
  - _base_
  - _self_

name: llama_cpp_llama

backend:
  device: mps
  model: TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF
  task: text-generation
  filename: tinyllama-1.1b-chat-v1.0.Q4_0.gguf

scenario:
  input_shapes:
    batch_size: 1
    sequence_length: 256
    vocab_size: 32000
  generate_kwargs:
    max_new_tokens: 100
    min_new_tokens: 100

Current limitations:

• Benchmarking is limited to a batch size of 1 due to limitations of the llama-cpp-python binding.
• MPS process isolation is not working (see issue Process isolation is not working on MPS device #230).
• No weights mode has not been implemented

Performance

The metrics were tested by comparing the results of the benchmark with a PyTorch backend and a llama.cpp backend. The performance results are close, and I can provide the full .json files if needed. Due to their low readability, they are not included here directly.

CLI output: (tested on M3 Pro CPU)

Performance with llama.cpp backend

Performance with the pytorch backend

The performance difference might be due to the significant amount of copying between devices with PyTorch, as shown below:

Furthermore, llama.cpp is optimized for Mac, which could explain the higher performance. Let me know if you want me to investigate the performance difference further

[IlyasMoutawwakil]

Thanks a lot ! Very awesome work !
Only missing test configs and GitHub workflows 🤗
Hopefully I fix the process launcher with mps by the time the mps workflows start running

[regisss]

Very clean PR!
I left a couple comments.

Additionally, since benchmarking for llama.cpp is limited to a batch size of 1, I think it may be good to add a comment right above the batch size field in the 3 example configs. Unless an error/warning is raised somewhere in the code but I didn't see it.

[baptistecolle]

Very clean PR! I left a couple comments.

Additionally, since benchmarking for llama.cpp is limited to a batch size of 1, I think it may be good to add a comment right above the batch size field in the 3 example configs. Unless an error/warning is raised somewhere in the code but I didn't see it.

It is done here
https://github.com/huggingface/optimum-benchmark/pull/231/files#r1690885402

[baptistecolle]

Thanks for the review. I implemented the necessary changes

I added:

• fix different formatting issue
• structure of the code, absolute import, forgotten debug print statement...)
• added support for embedding models with llama.cpp
• added tests and github workflow for llama.cpp backend

(also two of the runners are currently offline, so i am unable to run the CI on them)

• API ROCm Tests / build_image_and_run_api_rocm_tests (pull_request)
• CLI ROCm Pytorch Single-GPU Tests / run_cli_rocm_pytorch_single_gpu_tests (pull_request)

Let me know if you have more remarks

[IlyasMoutawwakil]

one or two examples are enough, there's a lot of repetition there

[baptistecolle]

Indeed, I created multiple config during development and forgot to remove it from the pr. I fixed it now

[IlyasMoutawwakil]

Great PR @baptistecolle 🤗