feat: Support nexa model with nexa-sdk #1053
Conversation
… not support start_server. not support other callings in nexa-sdk
There was a problem hiding this comment.
Thanks @LuciusMos , left some comments below, I think for now there's some issue which may make the model can't be called as expected, could you do some further test and add example? Thanks
camel/models/nexa_model.py
Outdated
|
|
||
| def __init__( | ||
| self, | ||
| model_type: ModelType, |
There was a problem hiding this comment.
there's no defined ModelType for Nexa
There was a problem hiding this comment.
Should I define ModelType for Nexa? Or should I just use str type for model platforms like Nexa as well as vLLM, ollama?
There was a problem hiding this comment.
now we support both ModelType and str as model_type input, please update with our latest master branch, I think for Nexa we can just use str since there are many models hosted
camel/models/nexa_model.py
Outdated
| response = self._client.chat.completions.create( | ||
| messages=messages, | ||
| # model=self.model_type, | ||
| model="model-type", |
There was a problem hiding this comment.
I think here shouldn't be hard coded
There was a problem hiding this comment.
Same question as above. This would be solved if I simply use str type.
camel/configs/nexa_config.py
Outdated
| the tokens with top_p probability mass. So :obj:`0.1` means only | ||
| the tokens comprising the top 10% probability mass are considered. | ||
| (default: :obj:`1.0`) | ||
| top_k (int, optional): The number of highest probability vocabulary |
There was a problem hiding this comment.
If we are going to use openai compatibility, I think this parameter would not be accepted
There was a problem hiding this comment.
Since there is no evidence that nexa-sdk officially supports OpenAI serving (but it works well in my testing), do you think it is ok for us to delete top_k and nctx for now, and support them in the future if there are updates in nexa-sdk (and openai)?
There was a problem hiding this comment.
from https://github.com/NexaAI/nexa-sdk they mentioned that they offers an OpenAI-compatible API server, and we can delete top_k and nctx for now
camel/configs/nexa_config.py
Outdated
| (default: :obj:`False`) | ||
| stop (str or list, optional): List of stop words for early stopping | ||
| generating further tokens. (default: :obj:`None`) | ||
| nctx (int, optional): Length of context window. (default: :obj:`2048`) |
There was a problem hiding this comment.
If we are going to use openai compatibility, I think this parameter would not be accepted
Wendong-Fan
changed the title
Description
Support
v1/chat/completionsMotivation and Context
close #925Types of changes
What types of changes does your code introduce? Put an
xin all the boxes that apply:Implemented Tasks
Checklist
Go over all the following points, and put an
xin all the boxes that apply.If you are unsure about any of these, don't hesitate to ask. We are here to help!