Query LLM

Query LLM is a simple, zero-dependency CLI tool for querying large language models (LLMs). It works seamlessly with both cloud-based LLM services (e.g., OpenAI GPT, Groq, OpenRouter) and locally hosted LLMs (e.g. llama.cpp, LM Studio, Ollama). Internally, it guides the LLM to perform step-by-step reasoning using the Chain of Thought method.

To run Query LLM, ensure that Node.js (v18 or higher) or Bun is installed.

./query-llm.js

To obtain quick responses, pipe a question directly:

echo "Top travel destinations in Indonesia?" | ./query-llm.js

For specific tasks:

echo "Translate 'thank you' into German" | ./query-llm.js

For simpler interactions with LLMs using zero-shot prompting, refer to the sister project, ask-llm.

Using Local LLM Servers

Supported local LLM servers include llama.cpp, Jan, Ollama, LocalAI, LM Studio, and Msty.

To utilize llama.cpp locally with its inference engine, load a quantized model like Llama-3.2 3B or Phi-3.5 Mini. Then set the LLM_API_BASE_URL environment variable:

/path/to/llama-server -m Llama-3.2-3B-Instruct-Q4_K_M.gguf
export LLM_API_BASE_URL=http://127.0.0.1:8080/v1

To use Jan with its local API server, refer to its documentation. Load a model like Llama-3.2 3B or Phi-3.5 Mini, and set the following environment variables:

export LLM_API_BASE_URL=http://127.0.0.1:1337/v1
export LLM_CHAT_MODEL='llama3-8b-instruct'

To use Ollama locally, load a model and configure the environment variable LLM_API_BASE_URL:

ollama pull llama3.2
export LLM_API_BASE_URL=http://127.0.0.1:11434/v1
export LLM_CHAT_MODEL='llama3.2'

For LocalAI, initiate its container and adjust the environment variable LLM_API_BASE_URL:

docker run -ti -p 8080:8080 localai/localai llama-3.2-3b-instruct:q4_k_m
export LLM_API_BASE_URL=http://localhost:3928/v1

For LM Studio, pick a model (e.g., Llama-3.2 3B). Next, go to the Developer tab, select the model to load, and click the Start Server button. Then, set the LLM_API_BASE_URL environment variable, noting that the server by default runs on port 1234:

export LLM_API_BASE_URL=http://127.0.0.1:1234/v1

For Msty, choose a model (e.g., Llama-3.2 3B) and ensure the local AI is running. Go to the Settings menu, under Local AI, and note the Service Endpoint (which defaults to port 10002). Then set the LLM_API_BASE_URL environment variable accordingly:

export LLM_API_BASE_URL=http://127.0.0.1:10002/v1

Using Managed LLM Services

Supported LLM services include AI21, Cerebras, Deep Infra, DeepSeek, Fireworks, Gemini, Groq, Hyperbolic, Lepton, Mistral, Nebius, Novita, OpenAI, OpenRouter, and Together.

For configuration specifics, refer to the relevant section. The quality of answers can vary based on the model's performance.

• AI21

export LLM_API_BASE_URL=https://api.ai21.com/studio/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL=jamba-1.5-mini

• Cerebras

export LLM_API_BASE_URL=https://api.cerebras.ai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="llama3.1-8b"

• Deep Infra

export LLM_API_BASE_URL=https://api.deepinfra.com/v1/openai
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Meta-Llama-3.1-8B-Instruct"

• DeepSeek

export LLM_API_BASE_URL=https://api.deepseek.com/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="deepseek-chat"

• Fireworks

export LLM_API_BASE_URL=https://api.fireworks.ai/inference/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="accounts/fireworks/models/llama-v3p1-8b-instruct"

• Google Gemini

export LLM_API_BASE_URL=https://generativelanguage.googleapis.com/v1beta
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="gemini-1.5-flash"

• Groq

export LLM_API_BASE_URL=https://api.groq.com/openai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="llama-3.1-8b-instant"

• Hyperbolic

export LLM_API_BASE_URL=https://api.hyperbolic.xyz/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Meta-Llama-3.1-8B-Instruct"

• Lepton

export LLM_API_BASE_URL=https://llama3-1-8b.lepton.run/api/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="llama3-1-8b"

• Mistral

export LLM_API_BASE_URL=https://api.mistral.ai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="open-mistral-7b"

• Nebius

export LLM_API_BASE_URL=https://api.studio.nebius.ai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Meta-Llama-3.1-8B-Instruct"

• Novita

export LLM_API_BASE_URL=https://api.novita.ai/v3/openai
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/llama-3.1-8b-instruct"

• OpenAI

export LLM_API_BASE_URL=https://api.openai.com/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="gpt-4o-mini"

• OpenRouter

export LLM_API_BASE_URL=https://openrouter.ai/api/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/llama-3.1-8b-instruct"

• Together

export LLM_API_BASE_URL=https://api.together.xyz/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"

Evaluating Questions

If there is a text file containing pairs of User and Assistant messages, it can be evaluated with Query LLM:

User: Which planet is the largest?
Assistant: The largest planet is /Jupiter/.

User: and the smallest?
Assistant: The smallest planet is /Mercury/.

Assuming the above content is in qa.txt, executing the following command will initiate a multi-turn conversation with the LLM, asking questions sequentially and verifying answers using regular expressions:

./query-llm.js qa.txt

For additional examples, please refer to the tests/ subdirectory.

Two environment variables can be used to modify the behavior:

• LLM_DEBUG_FAIL_EXIT: When set, Query LLM will exit immediately upon encountering an incorrect answer, and subsequent questions in the file will not be processed.

• LLM_DEBUG_PIPELINE: When set, and if the expected regular expression does not match the answer, the internal LLM pipeline will be printed to stdout.