Support fot local running LLM, like as LM studio

[ariael]

Hi,
Have you thought about supporting LM Studio as a local translation server? There are specialized LLMs available there. Possibly other frameworks for LLM, but LM studio looks the easiest. Local server is already included, it would just be a matter of extending the API support. I tried to adapt the current configuration options, but couldn't get the right answers to work.

The performance of the translation depends on the hardware, but on the other hand it is only about the cost of electricity and time.

[ariael]

"LLM Studio" support already OpenAI format for communications, i hope that change should be not much complicated...

[bookfere]

Thanks for your suggestion. You can use the custom engine to leverage LLMstudio for translation purposes.

{
    "name": "LLMstudio",
    "languages": {
        "source": {
            "German": "German",
            "Japanese": "Japanese"
        },
        "target": {
            "English": "English"
        }
    },
    "request": {
        "url": "http://localhost:8000/api/engine/chat/{provider}",
        "method": "POST",
        "headers": {"Content-Type": "application/json"},
        "data": {
            "api_key": "{api_key}",
            "model": "{model}",
            "chat_input": "Translate the content from <slang> to <tlang>: <text>",
        }
    },
    "response": "response['chat_output']"
}

Before using this code snippet, please replace {provider}, {api_key}, and {model} with the corresponding data first.

[ariael]

Hi,
i try your suggestion, this don't work.
I spend some time with debug needed format.
I have this:

{
    "name": "LLMstudio",
    "languages": {
        "source": {
            "English": "English"
        },
        "target": {
            "Czech": "Czech"
        }
    },
    "request": {
        "url": "http://localhost:1234/v1/chat/completions",
        "method": "POST",
        "headers": {"Content-Type": "application/json"},
        "data": {
            "messages": {"role": "translate", "content": "Translate the content from English to Czech: <text>"}
        }
    },
    "response": "response['chat_output']"
}

And response from server:

[2024-03-08 19:13:48.118] [INFO] [LM STUDIO SERVER] Verbose server logs are ENABLED
[2024-03-08 19:13:48.122] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 1234
[2024-03-08 19:13:48.123] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-03-08 19:13:48.123] [INFO] [LM STUDIO SERVER] ->	GET  http://localhost:1234/v1/models
[2024-03-08 19:13:48.124] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/chat/completions
[2024-03-08 19:13:48.124] [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/completions
[2024-03-08 19:13:48.125] [INFO] [LM STUDIO SERVER] Logs are saved into C:\tmp\lmstudio-server-log.txt
[2024-03-08 19:13:58.043] [INFO] [LM STUDIO SERVER] Processing queued request...
[2024-03-08 19:13:58.044] [INFO] Received POST request to /v1/chat/completions with body: {
  "messages": {
    "role": "translate",
    "content": "Translate the content from English to Czech: Hello World!"
  }
}
[2024-03-08 19:13:58.044] [ERROR] [Server Error] {"title":"'messages' field must be either an array of objects with 'role' and 'content' fields, or a string that can be parsed as JSON into an array of objects with 'role' and 'content' fields"}

[bookfere]

Could you provide the API documentation you refer to? I can check it for the code snippet you are using.

[ariael]

Hi,
I don't found separate document for LLM Studio, only mention "OpenAPI compatible"...
My work is from interactive work with response from local server.
On Discord LLM Studio i found this:

Load a model, start the server, and run this example in your terminal

Choose between streaming and non-streaming mode by setting the "stream" field

curl http://localhost:1234/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"messages": [
{ "role": "system", "content": "Always answer in rhymes." },
{ "role": "user", "content": "Introduce yourself." }
],
"temperature": 0.7,
"max_tokens": -1,
"stream": false
}'

Example: reuse your existing OpenAI setup

from openai import OpenAI

Point to the local server

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

completion = client.chat.completions.create(
model="local-model", # this field is currently unused
messages=[
{"role": "system", "content": "Always answer in rhymes."},
{"role": "user", "content": "Introduce yourself."}
],
temperature=0.7,
)

print(completion.choices[0].message)

Chat with an intelligent assistant in your terminal

from openai import OpenAI

Point to the local server

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

history = [
{"role": "system", "content": "You are an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful."},
{"role": "user", "content": "Hello, introduce yourself to someone opening this program for the first time. Be concise."},
]

while True:
completion = client.chat.completions.create(
model="local-model", # this field is currently unused
messages=history,
temperature=0.7,
stream=True,
)

new_message = {"role": "assistant", "content": ""}

for chunk in completion:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
        new_message["content"] += chunk.choices[0].delta.content

history.append(new_message)

# Uncomment to see chat history
# import json
# gray_color = "\033[90m"
# reset_color = "\033[0m"
# print(f"{gray_color}\n{'-'*20} History dump {'-'*20}\n")
# print(json.dumps(history, indent=2))
# print(f"\n{'-'*55}\n{reset_color}")

print()
history.append({"role": "user", "content": input("> ")})

---

Adapted from OpenAI's Vision example

from openai import OpenAI
import base64
import requests

Point to the local server

client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")

Ask the user for a path on the filesystem:

path = input("Enter a local filepath to an image: ")

Read the image and encode it to base64:

base64_image = ""
try:
image = open(path.replace("'", ""), "rb").read()
base64_image = base64.b64encode(image).decode("utf-8")
except:
print("Couldn't read the image. Make sure the path is correct and the file exists.")
exit()

completion = client.chat.completions.create(
model="local-model", # not used
messages=[
{
"role": "system",
"content": "This is a chat between a user and an assistant. The assistant is helping the user to describe an image.",
},
{
"role": "user",
"content": [
{"type": "text", "text": "What’s in this image?"},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
},
},
],
}
],
max_tokens=1000,
stream=True
)

for chunk in completion:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)

[bookfere]

If the input and output of the "LLM Studio" you used are the same as OpenAI's, you can try this snippet:

{
    "name": "LLMstudio",
    "languages": {
        "source": {
            "English": "English"
        },
        "target": {
            "Czech": "Czech"
        }
    },
    "request": {
        "url": "http://localhost:1234/v1/chat/completions",
        "method": "POST",
        "headers": {"Content-Type": "application/json"},
        "data": {
            "messages": [
                { "role": "system", "content": "Translate the content from <slang> to <tlang>." },
                { "role": "user", "content": "<text>" }
            ],
            "temperature": 0.7,
            "max_tokens": -1,
            "stream": false
        }
    },
    "response": "response['choices'][0]['message']['content']"
}

[ariael]

Thank you very much.
This looks much better.
In Calibre (EbookTanslator) is verify ok, and test result is without error. But without translation :)

Here is logs from LLM Studio

[2024-03-09 13:15:50.796] [INFO] [LM STUDIO SERVER] Verbose server logs are ENABLED
[2024-03-09 13:15:50.799] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 1234
[2024-03-09 13:15:50.799] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-03-09 13:15:50.800] [INFO] [LM STUDIO SERVER] -> GET http://localhost:1234/v1/models
[2024-03-09 13:15:50.800] [INFO] [LM STUDIO SERVER] -> POST http://localhost:1234/v1/chat/completions
[2024-03-09 13:15:50.800] [INFO] [LM STUDIO SERVER] -> POST http://localhost:1234/v1/completions
[2024-03-09 13:15:50.801] [INFO] [LM STUDIO SERVER] Logs are saved into C:\tmp\lmstudio-server-log.txt
[2024-03-09 13:16:21.518] [INFO] [LM STUDIO SERVER] Processing queued request...
[2024-03-09 13:16:21.519] [INFO] Received POST request to /v1/chat/completions with body: {
"messages": [
{
"role": "system",
"content": "Translate the content from to ."
},
{
"role": "user",
"content": "Hello World!"
}
],
"temperature": 0.7,
"max_tokens": -1,
"stream": false
}
[2024-03-09 13:16:21.519] [INFO] [LM STUDIO SERVER] Context Overflow Policy is: Truncate Middle
[2024-03-09 13:16:21.520] [INFO] [LM STUDIO SERVER] Last message: { role: 'user', content: 'Hello World!' } (total messages = 2)
[2024-03-09 13:16:22.005] [INFO] [LM STUDIO SERVER] Accumulating tokens ... (stream = false)
[2024-03-09 13:16:22.006] [INFO] Accumulated 1 tokens: Hello
[2024-03-09 13:16:22.067] [INFO] Accumulated 2 tokens: Hello World
[2024-03-09 13:16:22.139] [INFO] Accumulated 3 tokens: Hello World!
[2024-03-09 13:16:22.213] [INFO] Accumulated 4 tokens: Hello World!\n
[2024-03-09 13:16:22.287] [INFO] [LM STUDIO SERVER] Generated prediction: {
"id": "chatcmpl-h0wojk0kra2nr1b0gxg7c",
"object": "chat.completion",
"created": 1709986581,
"model": "C:\AI Models\TheBloke\CodeLlama-7B-Instruct-GGUF\codellama-7b-instruct.Q8_0.gguf",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello World!\n"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 4,
"completion_tokens": 4,
"total_tokens": 8
}
}

[ariael]

And this version translate test call in Ebook translator

{
    "name": "LLMstudio",
    "languages": {
        "source": {
            "English": "English"
        },
        "target": {
            "Czech": "Czech"
        }
    },
    "request": {
        "url": "http://localhost:1234/v1/chat/completions",
        "method": "POST",
        "headers": {"Content-Type": "application/json"},
        "data": {
            "messages": [
                { "role": "system", "content": "Translate the content from English to Czech." },
                { "role": "user", "content": "<text>" }
            ],
            "temperature": 0.7,
            "max_tokens": -1,
            "stream": false
        }
    },
    "response": "response['choices'][0]['message']['content']"
}

[ariael]

Yea, results is much better, but...
Lots of parse error - i try use 30 second wait ant results is better.
Sometimes LLM halucinate :)
And worse - result book is not translated. Some small error somewhere...

lmstudio-server-log.txt