Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Qwen handler and fix mean_latency calculation error for OSS models #642

Merged
merged 8 commits into from
Oct 5, 2024

Conversation

zhangch-ss
Copy link
Contributor

Hello, there may be an anomaly in the mean latency calculation of the OSS model, and I have attempted to fix it.

@zhangch-ss
Copy link
Contributor Author

line 1045, berkeley-function-call-leaderboard/bfcl/eval_checker/eval_runner_helper.py

@zhangch-ss
Copy link
Contributor Author

Qwen handler is also added in this PR
image

@zhangch-ss zhangch-ss changed the title Fix the mean_latency calculation error in the OSS model Add Qwen handler and fix mean_latency calculation error for OSS models Sep 19, 2024
@zhangch-ss
Copy link
Contributor Author

You may need to modify it according to the actual situation

"Qwen/Qwen2-1.5B-Instruct": 100,
"Qwen/Qwen2-7B-Instruct": 100,
OSS_LATENCY = {
    "deepseek-ai/deepseek-coder-6.7b-instruct": 909,
    "google/gemma-7b-it": 95,
    "NousResearch/Hermes-2-Pro-Mistral-7B": 135,
    "NousResearch/Hermes-2-Pro-Llama-3-8B": 77,
    "NousResearch/Hermes-2-Theta-Llama-3-8B": 73,
    "NousResearch/Hermes-2-Theta-Llama-3-70B": 716,
    "NousResearch/Hermes-2-Pro-Llama-3-70B": 674,
    "meta-llama/Meta-Llama-3-8B-Instruct": 73,
    "meta-llama/Meta-Llama-3-70B-Instruct": 307,
    "gorilla-openfunctions-v2": 83,
    "THUDM/glm-4-9b-chat": 223,
    "Qwen/Qwen2-1.5B-Instruct": 100,
    "Qwen/Qwen2-7B-Instruct": 100,
}

@zhangch-ss
Copy link
Contributor Author

Now it supports the newly released Qwen2.5
image

@HuanzhiMao
Copy link
Collaborator

Apologize for the long delay. Will definitely take a look after the Monday ICLR deadline.

@zhangch-ss
Copy link
Contributor Author

Apologize for the long delay. Will definitely take a look after the Monday ICLR deadline.

thanks, hope all is well

Copy link
Collaborator

@HuanzhiMao HuanzhiMao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @zhangch-ss and looking forward to the amazing Qwen models.
I made a few changes to your PR so that the handler is compatible with the latest codebase.
Regarding the mean/latency calculation, I will update those in a separate PR, as that involves quite a bit of change.

@HuanzhiMao HuanzhiMao added the BFCL-New Model Add New Model to BFCL label Oct 5, 2024
@ShishirPatil ShishirPatil merged commit b4edd01 into ShishirPatil:main Oct 5, 2024
ShishirPatil pushed a commit that referenced this pull request Oct 21, 2024
This PR updates the leaderboard to reflect the change in score due to
the following PR merge:

1. #660 
2. #661
3. #683
4. #679
5. #708 
6. #709
7. #701
8. #657 
9. #658 
10. #640 
11. #653
12. #642 
13. #696 
14. #667

Close #662.

Note: Some models (like `firefunction`, `functionary`,
`microsoft/phi`)are not included in this leaderboard update because we
don't have all the entries generated. We will add them back once we get
the full result generated.
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
ShishirPatil#642)

Hello, there may be an anomaly in the mean latency calculation of the
OSS model, and I have attempted to fix it.

---------

Co-authored-by: ai_user <ai@digitalchina.com>
Co-authored-by: Huanzhi (Hans) Mao <huanzhimao@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFCL-New Model Add New Model to BFCL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants