mlx-engine - Apple MLX LLM Engine for LM Studio

mlx-engine

MLX engine for LM Studio

Built with

• mlx-lm - Apple MLX inference engine (MIT)
• Outlines - Structured output for LLMs (Apache 2.0)
• mlx-vlm - Vision model inferencing for MLX (MIT)

How to use in LM Studio

LM Studio 0.3.4 and newer for Mac ships pre-bundled with mlx-engine. Download LM Studio from here

Standalone Demo

Install Steps

To run a demo of model load and inference:

1. Clone the repository

git clone https://github.com/lmstudio-ai/mlx-engine.git
cd mlx-engine

2. Create a virtual environment (optional)

 python -m venv myenv
 source myenv/bin/activate

3. Install the required dependency packages

pip install -r requirements.txt

Text Model Demo

Run the demo.py script with an MLX text model:

python demo.py --model ~/.cache/lm-studio/models/mlx-community/Meta-Llama-3.1-8B-Instruct-4bit 

mlx-community/Meta-Llama-3.1-8B-Instruct-4bit - 4.53 GB

This command will use a default prompt that is formatted for Llama-3.1. For other models, add a custom --prompt argument with the correct prompt formatting:

python demo.py --model ~/.cache/lm-studio/models/mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx --prompt "<｜begin▁of▁sentence｜>User: How long will it take an apple to fall from a 10m tree?Assistant:"

mlx-community/DeepSeek-Coder-V2-Lite-Instruct-4bit-mlx - 8.84 GB

Vision Model Demo

Run the demo.py script with an MLX vision model:

python demo.py --model ~/.cache/lm-studio/models/mlx-community/pixtral-12b-4bit --prompt "<s>[INST]Compare these images[IMG][IMG][/INST]" --images demo-data/chameleon.webp demo-data/toucan.jpeg

Currently supported vision models and download links:

• Pixtral
  • mlx-community/pixtral-12b-4bit - 7.15 GB
• Qwen2-VL
  • mlx-community/Qwen2-VL-2B-4bit - 1.26 GB
  • mlx-community/Qwen2-VL-7B-4bit - 4.68 GB
• Llava-v1.6
  • mlx-community/llava-v1.6-mistral-7b-4bit - 4.26 GB