Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[InferenceClient] flag chat_completion()'s logit_bias as UNUSED #2724

Merged
merged 3 commits into from
Jan 6, 2025

Conversation

hanouticelina
Copy link
Contributor

Related to #2720.
logit_bias is not implemented in TGI (see the comment here), this PR updates the parameter documentation to flag it as unused.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@juanwisz
Copy link

Could we consider actually implementing logit_bias? It's a very valuable feature for controlling model outputs, I've opened in TGI to discuss implementation details: huggingface/text-generation-inference#2869

@hanouticelina
Copy link
Contributor Author

Could we consider actually implementing logit_bias? It's a very valuable feature for controlling model outputs, I've opened in TGI to discuss implementation details: huggingface/text-generation-inference#2869

@juanwisz This PR only aligns the client documentation with the current TGI implementation. Once/If logit_bias is implemented on TGI side, we'll update the documentation accordingly 🤗

@LysandreJik
Copy link
Member

Is it possible to remove that doc attribute altogether? If not, I would at least specify why it's unused and how to have users request for this argument to be used

@hanouticelina
Copy link
Contributor Author

Is it possible to remove that doc attribute altogether? If not, I would at least specify why it's unused and how to have users request for this argument to be used

@LysandreJik logit_bias and n (i.e number of chat completion choices to generate for each input message) which is already flagged as unused, are implemented in the OpenAI API, and just like we expose these parameters in TGI to maintain OpenAI compatibility, we want to keep them in the client for the same reason —making InferenceClient a proper drop-in replacements for OpenAI client. I've updated their documentation to indicate they are currently not implemented in TGI while explaining why they are still included.

Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok great! Thank you Célina !

@hanouticelina hanouticelina merged commit 6f5d870 into main Jan 6, 2025
17 checks passed
@hanouticelina hanouticelina deleted the update-logit-bias-documentation branch January 6, 2025 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants