Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BFCL] Add ToolACE handler for BFCL-v3 #653

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

XuHwang
Copy link

@XuHwang XuHwang commented Sep 23, 2024

This PR adds the handler of the ToolACE model, which finetunes LLaMA-3.1-8B-Instruct model with ToolACE dataset, obtaining wonderful points in functional calling.

We have adapted our handler compatible with version 3.

Here are the results of the version evaluated in our machine (4*v100-32GB). We also found that the results would change in different machines.

Rank Overall Acc Non-Live AST Acc Non-Live Simple AST Non-Live Multiple AST Non-Live Parallel AST Non-Live Parallel Multiple AST Non-Live Exec Acc Non-Live Simple Exec Non-Live Multiple Exec Non-Live Parallel Exec Non-Live Parallel Multiple Exec Live Acc Live Simple AST Live Multiple AST Live Parallel AST Live Parallel Multiple AST Multi Turn Acc Multi Turn Base Multi Turn Miss Func Multi Turn Miss Param Multi Turn Long Context Multi Turn Composite Relevance Detection Irrelevance Detection
1 59.22% 89.27% 80.58% 95.00% 91.00% 90.50% 90.07% 98.29% 94.00% 88.00% 80.00% 73.21% 62.79% 74.25% 81.25% 75.00% 14.37% 21.50% 6.50% 17.50% 12.00% N/A 85.37% 83.81%

Thanks for your efforts in holding such a wonderful leaderboard. We need your help (@HuanzhiMao, @CharlieJCJ ) in adding our model to the leaderboard. Thanks a lot~

@XuHwang XuHwang mentioned this pull request Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant