Multi-shot how to?

[nigel-daniels]

Hi, if I want to set up a multi-shot prompt how should I construct the prompt? Or would thus need a new feature to allow the submission of an array of chat prompts?

[giladgd]

I think the best approach for you to do that right now is to create multiple LlamaContexts with a LlamaChatSession for each one, and then load all the history into all of the chat sessions, so you can treat the response of each session as a different option.
I'll see if I can make it possible to clone a context to keep its current state for improved performance of this operation

[nigel-daniels]

Hi @giladgd thanks for the answer :) As some context for the question, I was thinking of implementing a langchainjs ChatLLM module using node-llama-cpp as I did for a base LLM. If you're considering a change to support this, would it make sense to pass the LlamaChatSession an array of ChatPromptWrappers? I could transform the langchainjs prompts to node-llama-cpp prompts during the construction.

[giladgd]

Hi @nigel-daniels, you're welcome :)
It would be awesome if you could implement a langchainjs ChatLLM module for node-llama-cpp :)
I suggest you try to incorporate the JSON schema grammar of node-llama-cpp for the function interfaces, as I think it can help make a model generate valid outputs without too much fine-tunings.

I'm not sure I understand what you are trying to accomplish with the ChatPromptWrappers.
Are you trying to prompt a model the same thing but with a different ChatPromptWrappers to see how the model reacts to each one?

Ideally, each model will have only one ChatPromptWrapper made for it, so I don't see what value would you get from trying to prompt it with a different ChatPromptWrapper.
For example, if you chat with a LLama Chat model using the ChatMLChatPromptWrapper, it may work, but since it wasn't designed for it, I wouldn't want to use it for any application.

I tried implementing the ability to clone a LlamaChatSession object together with its LlamaContext to be able to "fork" the current context state, so it would be possible to generate multiple responses from the same original state without loading the entire state again every time.
Unfortunately, there's a bug in llama.cpp that makes it impossible to do ATM.
I've opened an issue about it, and will try to make it work after the issue is fixed: ggerganov/llama.cpp#3606

[nigel-daniels]

The reason I was asking aboutChatPromptWrapper is the main method to implement for a ChatLLM is the call method, this is typically passed an array of messages, e.g. system message, human message or AI message. Typically just a human message is provided (i.e. the prompt). Generally at the start of a session you can see a system message and a human message. I can see how to intercept a system message and instantiate a session with that set. Thinking about it if more messages are passed I can just concatenate them into a prompt with the appropriate tags. So long as I respond with an AI message all should be good.

[stduhpf]

Hi @giladgd. I see that the issue in question (ggerganov/llama.cpp#3606) was closed as completed quite some time ago on llama.cpp repo. Does that mean cloning LlamaContext works now? If so, how?

[giladgd]

@stduhpf Adding this ability as-is is a bit problematic as this feature doesn't play well with other features and requires a deep understanding of the implementation details on the llama.cpp side just to use it without affecting unrelated things, so eventually, I don't plan to add support for it as-is soon.
I aim to keep node-llama-cpp as high-level as possible, so there will be no need to delve into the implementation details of llama.cpp to use it.

However, one of the next beta versions will add a new feature that I think will provide a sufficient alternative while still being efficient.
This new feature will utilize the identical part of the state of disposed context sequences for new evaluations, so when you try to evaluate some tokens on a "clean" context sequence, and then try to evaluate some other tokens on a "clean" context sequence and those tokens start with some part the is the same as the previous evaluation, the result from the previous evaluation for this part will be reused to save you some evaluation time.