Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama-cpp-server broken #547

Open
jeffmaury opened this issue Jun 11, 2024 · 7 comments · Fixed by #565
Open

llama-cpp-server broken #547

jeffmaury opened this issue Jun 11, 2024 · 7 comments · Fixed by #565
Labels
bug Something isn't working

Comments

@jeffmaury
Copy link
Collaborator

Got this while running from main branch in Podman AI Lab:

�llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /granite-7b-lab-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = launch-day
llama_model_loader: - kv   2:                           llama.vocab_size u32              = 32008
llama_model_loader: - kv   3:                       llama.context_length u32              = 2048
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   5:                          llama.block_count u32              = 32
llama_model_loader: - kv   6:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   7:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   8:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   9:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv  10:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  12:                          general.file_type u32              = 15
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = llama
�llama_model_loader: - kv  14:                      tokenizer.ggml.tokens arr[str,32008]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
�llama_model_loader: - kv  15:                      tokenizer.ggml.scores arr[f32,32008]   = [0.000000, 0.000000, 0.000000, 0.0000...
�llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,32008]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  18:                tokenizer.ggml.eos_token_id u32              = 32000
llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 32001
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  22:               tokenizer.ggml.add_eos_token bool             = false
�llama_model_loader: - kv  23:                    tokenizer.chat_template str              = {% for message in messages %}{% if me...
llama_model_loader: - kv  24:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
llm_load_vocab: special tokens definition check successful ( 267/32008 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32008
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 2048
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 2048
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.80 GiB (4.84 BPW) 
llm_load_print_meta: general.name     = launch-day
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 32000 '<|endoftext|>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 32001 '<|pad|>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_print_meta: EOT token        = 32000 '<|endoftext|>'
llm_load_tensors: ggml ctx size =    0.15 MiB
llm_load_tensors:        CPU buffer size =  3891.29 MiB
warning: failed to mlock 74489856-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
..................................................................................................
llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.14 MiB
llama_new_context_with_model:        CPU compute buffer size =   164.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 1
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
�Model metadata: {'tokenizer.chat_template': "{% for message in messages %}{% if message['role'] == 'system' %}{{'<|system|>'+ '\n' + message['content'] + '\n'}}{% elif message['role'] == 'user' %}{{'<|user|>' + '\n' + message['content'] + '\n'}}{% elif message['role'] == 'assistant' %}{{'<|assistant|>' + '\n' + message['content'] + '<|endoftext|>' + ('' if loop.last else '\n')}}{% endif %}{% endfor %}", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.padding_token_id': '32001', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '32000', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '2048', 'general.name': 'launch-day', 'llama.vocab_size': '32008', 'general.file_type': '15', 'tokenizer.ggml.add_bos_token': 'false', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '11008', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '32'}
Available chat formats from metadata: chat_template.default
INFO:     Started server process [2]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
INFO:     127.0.0.1:44284 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:44286 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:49090 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:49092 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:51242 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:47010 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:47018 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34398 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:34412 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:53540 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57092 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:57104 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:44972 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:44974 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:46726 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:53006 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:53008 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40858 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:40860 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:43442 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:54454 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:54462 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:54468 - "GET /api/tags HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:54480 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:56162 - "GET / HTTP/1.1" 404 Not Found
�Exception: Invalid chat handler: None (valid formats: ['llama-2', 'llama-3', 'alpaca', 'qwen', 'vicuna', 'oasst_llama', 'baichuan-2', 'baichuan', 'openbuddy', 'redpajama-incite', 'snoozy', 'phind', 'intel', 'open-orca', 'mistrallite', 'zephyr', 'pygmalion', 'chatml', 'mistral-instruct', 'chatglm3', 'openchat', 'saiga', 'gemma', 'functionary', 'functionary-v2', 'functionary-v1', 'chatml-function-calling'])
Traceback (most recent call last):
�  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 127, in get_chat_completion_handler_by_name
    chat_handler = self._chat_handlers[name]
                   ~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'None'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/errors.py", line 171, in custom_route_handler
    response = await original_route_handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/app.py", line 483, in create_chat_completion
    ] = await run_in_threadpool(llama.create_chat_completion, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama.py", line 1728, in create_chat_completion
    handler = self.chat_handler or self._chat_handlers.get(self.chat_format) or llama_chat_format.get_chat_completion_handler(
                                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 136, in get_chat_completion_handler
    return LlamaChatCompletionHandlerRegistry().get_chat_completion_handler_by_name(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 130, in get_chat_completion_handler_by_name
    raise LlamaChatCompletionHandlerNotFoundException(
�llama_cpp.llama_chat_format.LlamaChatCompletionHandlerNotFoundException: Invalid chat handler: None (valid formats: ['llama-2', 'llama-3', 'alpaca', 'qwen', 'vicuna', 'oasst_llama', 'baichuan-2', 'baichuan', 'openbuddy', 'redpajama-incite', 'snoozy', 'phind', 'intel', 'open-orca', 'mistrallite', 'zephyr', 'pygmalion', 'chatml', 'mistral-instruct', 'chatglm3', 'openchat', 'saiga', 'gemma', 'functionary', 'functionary-v2', 'functionary-v1', 'chatml-function-calling'])
INFO:     127.0.0.1:56172 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
�Exception: Invalid chat handler: None (valid formats: ['llama-2', 'llama-3', 'alpaca', 'qwen', 'vicuna', 'oasst_llama', 'baichuan-2', 'baichuan', 'openbuddy', 'redpajama-incite', 'snoozy', 'phind', 'intel', 'open-orca', 'mistrallite', 'zephyr', 'pygmalion', 'chatml', 'mistral-instruct', 'chatglm3', 'openchat', 'saiga', 'gemma', 'functionary', 'functionary-v2', 'functionary-v1', 'chatml-function-calling'])
Traceback (most recent call last):
�  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 127, in get_chat_completion_handler_by_name
    chat_handler = self._chat_handlers[name]
                   ~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'None'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/errors.py", line 171, in custom_route_handler
    response = await original_route_handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/app.py", line 483, in create_chat_completion
    ] = await run_in_threadpool(llama.create_chat_completion, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama.py", line 1728, in create_chat_completion
    handler = self.chat_handler or self._chat_handlers.get(self.chat_format) or llama_chat_format.get_chat_completion_handler(
                                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 136, in get_chat_completion_handler
    return LlamaChatCompletionHandlerRegistry().get_chat_completion_handler_by_name(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 130, in get_chat_completion_handler_by_name
    raise LlamaChatCompletionHandlerNotFoundException(
�llama_cpp.llama_chat_format.LlamaChatCompletionHandlerNotFoundException: Invalid chat handler: None (valid formats: ['llama-2', 'llama-3', 'alpaca', 'qwen', 'vicuna', 'oasst_llama', 'baichuan-2', 'baichuan', 'openbuddy', 'redpajama-incite', 'snoozy', 'phind', 'intel', 'open-orca', 'mistrallite', 'zephyr', 'pygmalion', 'chatml', 'mistral-instruct', 'chatglm3', 'openchat', 'saiga', 'gemma', 'functionary', 'functionary-v2', 'functionary-v1', 'chatml-function-calling'])
INFO:     127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     127.0.0.1:56196 - "GET / HTTP/1.1" 404 Not Found
�Exception: Invalid chat handler: None (valid formats: ['llama-2', 'llama-3', 'alpaca', 'qwen', 'vicuna', 'oasst_llama', 'baichuan-2', 'baichuan', 'openbuddy', 'redpajama-incite', 'snoozy', 'phind', 'intel', 'open-orca', 'mistrallite', 'zephyr', 'pygmalion', 'chatml', 'mistral-instruct', 'chatglm3', 'openchat', 'saiga', 'gemma', 'functionary', 'functionary-v2', 'functionary-v1', 'chatml-function-calling'])
Traceback (most recent call last):
�  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 127, in get_chat_completion_handler_by_name
    chat_handler = self._chat_handlers[name]
                   ~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'None'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/errors.py", line 171, in custom_route_handler
    response = await original_route_handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/server/app.py", line 483, in create_chat_completion
    ] = await run_in_threadpool(llama.create_chat_completion, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama.py", line 1728, in create_chat_completion
    handler = self.chat_handler or self._chat_handlers.get(self.chat_format) or llama_chat_format.get_chat_completion_handler(
                                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 136, in get_chat_completion_handler
    return LlamaChatCompletionHandlerRegistry().get_chat_completion_handler_by_name(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�  File "/opt/app-root/lib64/python3.11/site-packages/llama_cpp/llama_chat_format.py", line 130, in get_chat_completion_handler_by_name
    raise LlamaChatCompletionHandlerNotFoundException(
�llama_cpp.llama_chat_format.LlamaChatCompletionHandlerNotFoundException: Invalid chat handler: None (valid formats: ['llama-2', 'llama-3', 'alpaca', 'qwen', 'vicuna', 'oasst_llama', 'baichuan-2', 'baichuan', 'openbuddy', 'redpajama-incite', 'snoozy', 'phind', 'intel', 'open-orca', 'mistrallite', 'zephyr', 'pygmalion', 'chatml', 'mistral-instruct', 'chatglm3', 'openchat', 'saiga', 'gemma', 'functionary', 'functionary-v2', 'functionary-v1', 'chatml-function-calling'])
INFO:     127.0.0.1:56204 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     127.0.0.1:52002 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:56736 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:56742 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:55332 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:55338 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:37208 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:37220 - "GET / HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:49620 - "GET / HTTP/1.1" 404 Not Found
@jeffmaury
Copy link
Collaborator Author

See my comment: #539 (comment)

@lmilbaum
Copy link
Collaborator

@MichaelClifford @Gregory-Pereira Is that something you can look at? It would also make sense to implement a CI/Testing Framework test. WDYT?

@cooktheryan cooktheryan added the bug Something isn't working label Jun 13, 2024
@Gregory-Pereira
Copy link
Collaborator

I will take a look at this tonight

@cooktheryan
Copy link
Collaborator

@Gregory-Pereira do you want me to take this one over?

@Gregory-Pereira
Copy link
Collaborator

The temporary fix is to add back the chat_format to the makefile for only runs or to the run.sh but this invalidates the work that @jeffmaury wanted around check the chat_format from the model first. The answer to this is that we cannot leverage the native llamacpp functionality to do this, we could add something to the run.sh to do this in a smarter way (ie read the .gguf model metadata from the $MODEL_PATH we already have and propgate it as the value to the run.sh script... Ill need to think on this. For the meantime il put a temp fix in

@Gregory-Pereira Gregory-Pereira mentioned this issue Jun 14, 2024
Merged
@Gregory-Pereira
Copy link
Collaborator

Maybe this might be usable in this smarter implementation: https://github.com/ggerganov/llama.cpp/blob/master/gguf-py/scripts/gguf-new-metadata.py

@cooktheryan cooktheryan reopened this Jun 14, 2024
@cooktheryan
Copy link
Collaborator

Reopened to keep this here for the long term fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

temp fix revert to llama-2 format by default containers/ai-lab-recipes
4 participants
@jeffmaury @cooktheryan @lmilbaum @Gregory-Pereira