LLMatic can be used as a drop-in replacement for OpenAI's API (see the supported endpoints). It uses llama-node with llama.cpp backend to run the models locally.
Supported endpoints:
- /completions (stream and non-stream)
- /chat/completions (stream and non-stream)
- /embeddings
- /models
This project is currently a work in progress. At this point, it's recommended to use it only for ad-hoc development and testing.
I'm looking for contributors to help me with the open issues. If you're interested, please leave a comment on the issue you want to work on.
Also, if you have any good ideas for improving this project, please open an issue to discuss it further.
The main motivation behind making LLMatic was to experiment with OpenAI's API without worrying about the cost. I have seen other attempts at creating OpenAI-Compatible APIs such as:
But I wanted a small, simple, and easy to extend implementation in TypeScript based on the official OpenAI API specification.
If you prefer a video tutorial, you can watch the following video for step-by-step instructions on how to use this project:
- Node.js >=18.16
- Unix-based OS (Linux, macOS, WSL, etc.)
Create an empty directory and run npm init:
export LLMATIC_PROJECT_DIR=my-llmatic-project
mkdir $LLMATIC_PROJECT_DIR
cd $LLMATIC_PROJECT_DIR
npm init -yInstall and configure LLMatic:
npm add llmatic
# Download a model and generate a config file
npx llmatic configAdjust the config file to your needs and start the server:
npx llmatic startYou can run llmatic --help to see all available commands.
Usage with chatbot-ui
Clone the repo and install the dependencies:
git clone https://github.com/mckaywrigley/chatbot-ui.git
cd chatbot-ui
npm installCreate a .env.local file:
cat <<EOF > .env.local
# For now, this is ignored by LLMatic
DEFAULT_MODEL=Ignored
NEXT_PUBLIC_DEFAULT_SYSTEM_PROMPT=A chat between a curious human (user) and an artificial intelligence assistant (assistant). The assistant gives helpful, detailed, and polite answers to the human's questions.
user: Hello!
assistant: Hello! How may I help you today?
user: Please tell me the largest city in Europe.
assistant: Sure. The largest city in Europe is Moscow, the capital of Russia.
OPENAI_API_KEY=ANYTHING_WILL_DO
OPENAI_API_HOST=http://localhost:3000
GOOGLE_API_KEY=YOUR_API_KEY
GOOGLE_CSE_ID=YOUR_ENGINE_ID
EOFRun the server:
npm run dev -- --port 3001Demo:
Usage with LangChain
There are two examples of using LLMatic with LangChain in the
examples directory.
To run the Node.js example, first install the dependencies:
cd examples/node-langchain
npm installThen run the main script:
npm startExpand this to see the sample output
[chain/start] [1:chain:llm_chain] Entering Chain run with input: {
"humanInput": "Rememeber that this is a demo of LLMatic with LangChain.",
"history": ""
}
[llm/start] [1:chain:llm_chain > 2:llm:openai] Entering LLM run with input: {
"prompts": [
"A chat between a curious user and an artificial intelligence assistant.\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\n\n\nHuman: Rememeber that this is a demo of LLMatic with LangChain.\nAI:"
]
}
[llm/end] [1:chain:llm_chain > 2:llm:openai] [5.92s] Exiting LLM run with output: {
"generations": [
[
{
"text": " Yes, I understand. I am ready to assist you with your queries.",
"generationInfo": {
"finishReason": "stop",
"logprobs": null
}
}
]
],
"llmOutput": {
"tokenUsage": {}
}
}
[chain/end] [1:chain:llm_chain] [5.92s] Exiting Chain run with output: {
"text": " Yes, I understand. I am ready to assist you with your queries."
}
[chain/start] [1:chain:llm_chain] Entering Chain run with input: {
"humanInput": "What did I ask you to remember?",
"history": "Human: Rememeber that this is a demo of LLMatic with LangChain.\nAI: Yes, I understand. I am ready to assist you with your queries."
}
[llm/start] [1:chain:llm_chain > 2:llm:openai] Entering LLM run with input: {
"prompts": [
"A chat between a curious user and an artificial intelligence assistant.\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\n\nHuman: Rememeber that this is a demo of LLMatic with LangChain.\nAI: Yes, I understand. I am ready to assist you with your queries.\nHuman: What did I ask you to remember?\nAI:"
]
}
[llm/end] [1:chain:llm_chain > 2:llm:openai] [6.51s] Exiting LLM run with output: {
"generations": [
[
{
"text": " You asked me to remember that this is a demo of LLMatic with LangChain.",
"generationInfo": {
"finishReason": "stop",
"logprobs": null
}
}
]
],
"llmOutput": {
"tokenUsage": {}
}
}
[chain/end] [1:chain:llm_chain] [6.51s] Exiting Chain run with output: {
"text": " You asked me to remember that this is a demo of LLMatic with LangChain."
}
To run the Python example, first install the dependencies:
cd examples/python-langchain
pip3 install -r requirements.txtThen run the main script:
python3 main.pyExpand this to see the sample output
> Entering new LLMChain chain...
Prompt after formatting:
A chat between a curious user and an artificial intelligence assistant.
The assistant gives helpful, detailed, and polite answers to the user's questions.
Human: Rememeber that this is a demo of LLMatic with LangChain.
AI:
> Finished chain.
Yes, I understand. I am ready to assist you with your queries.
> Entering new LLMChain chain...
Prompt after formatting:
A chat between a curious user and an artificial intelligence assistant.
The assistant gives helpful, detailed, and polite answers to the user's questions.
Human: Rememeber that this is a demo of LLMatic with LangChain.
AI: Yes, I understand. I am ready to assist you with your queries.
Human: What did I ask you to remember?
AI:
> Finished chain.
You asked me to remember that this is a demo of LLMatic with LangChain.
LLMatic is designed to be easily extensible. You can create your own adapters by extending the LlmAdapter class. See examples/custom-adapter for an example.
To start llmatic with a custom adapter, use the --llm-adapter flag:
llmatic start --llm-adapter ./custom-llm-adapter.ts