-
Notifications
You must be signed in to change notification settings - Fork 996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Qwen handler and fix mean_latency calculation error for OSS models #642
Conversation
line 1045, berkeley-function-call-leaderboard/bfcl/eval_checker/eval_runner_helper.py |
You may need to modify it according to the actual situation
|
Apologize for the long delay. Will definitely take a look after the Monday ICLR deadline. |
thanks, hope all is well |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @zhangch-ss and looking forward to the amazing Qwen models.
I made a few changes to your PR so that the handler is compatible with the latest codebase.
Regarding the mean/latency calculation, I will update those in a separate PR, as that involves quite a bit of change.
This PR updates the leaderboard to reflect the change in score due to the following PR merge: 1. #660 2. #661 3. #683 4. #679 5. #708 6. #709 7. #701 8. #657 9. #658 10. #640 11. #653 12. #642 13. #696 14. #667 Close #662. Note: Some models (like `firefunction`, `functionary`, `microsoft/phi`)are not included in this leaderboard update because we don't have all the entries generated. We will add them back once we get the full result generated.
ShishirPatil#642) Hello, there may be an anomaly in the mean latency calculation of the OSS model, and I have attempted to fix it. --------- Co-authored-by: ai_user <ai@digitalchina.com> Co-authored-by: Huanzhi (Hans) Mao <huanzhimao@gmail.com>
Hello, there may be an anomaly in the mean latency calculation of the OSS model, and I have attempted to fix it.