llama.cpp / GGUF support

[sammcj]

It would be great to see OLMoE/OlmoeForCausalLM Llama.cpp/GGUF support.

Really neat project!

[AmitKKhanchandani]

+1, need to try this with ollama :)

[Bobetele]

yes

[Muennighoff]

won't have bandwidth to do this, but if anyone is interested, that'd be amazing!

[MrDowntempo]

Yeah, this is hard to work with if it isn't in GGUF format to run locally, or available from Ollama directly. I'm looking into how to serve from Safetensors but not a lot of servers support that.

[Muennighoff]

also cc @2015aroras

[2015aroras]

See ggerganov/llama.cpp#9462

[Meshwa428]

See ggerganov/llama.cpp#9462

It still isn't merged 😞