Add ToolACE handler

[XuHwang]

This PR adds the handler of the ToolACE model, which finetunes LLaMA-3.1-8B-Instruct model with ToolACE dataset, obtaining wonderful points in functional calling.

Here are the results on the latest (0824) version evaluated in our machine:

Rank	Overall Acc	Model	AST Summary	Exec Summary	Simple AST	Multiple AST	Parallel AST	Parallel Multiple AST	Simple Exec	Multiple Exec	Parallel Exec	Parallel Multiple Exec	Irrelevance	Relevance
1	85.91%	ToolACE-8B (FC)	80.24%	92.21%	70.76%	85.78%	84.00%	80.42%	96.86%	94.00%	88.00%	90.00%	81.50%	87.80%

Thanks for your efforts in holding such a wonderful leaderboard. We need your help (@HuanzhiMao, @CharlieJCJ ) in adding our model to the leaderboard. Thanks a lot～

[HuanzhiMao]

Thank you for the PR and welcome! We’re currently busy with a new dataset release. I’ll review your submission later this week and aim to provide feedback by next Monday. Apologies for the delay.

[HuanzhiMao]

Hey @XuHwang, one quick question: Where can I find the documentation of this model? Eg, it's input schema, prompt format, etc.

[XuHwang]

Hey @XuHwang, one quick question: Where can I find the documentation of this model? Eg, it's input schema, prompt format, etc.

Thanks for the reply.

The format of prompt is actually given in the handler, which is similar to the prompt of LLaMA3. The format should be:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{SYSTEM_PROMPT}{TOOL_DESC}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{QUERY}\n{LANGUAGE_SPECIFIC_HINT}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n

[HuanzhiMao]

So why can't we just use the LlamaHandler instead? The ToolACEHandler has the exact same _format_prompt, decode_ast, decode_execute, etc as the LlamaHandler.

[XuHwang]

So why can't we just use the LlamaHandler instead? The ToolACEHandler has the exact same _format_prompt, decode_ast, decode_execute, etc as the LlamaHandler.

Actually the decode_ast, decode_execute are consistent with the LlamaHandler, but the _format_prompt and process_input are slightly different, and we find it would cause obvious performance degradation.

[HuanzhiMao]

Compared with the current latest LlamaHandler from main branch commit eca516a, your ToolACEHandler differs in the following two ways:

1. Your _format_prompt has two more \n after each <|end_header_id|> tag. This is an oversight on our end; we missed it for the Llama models. I have just fixed it in [BFCL] Fix Llama Handler #626. After the fix, they should be the exact same.
2. You choose to add the language_specific_hint to the end of the last user prompt instead of as part of the function description in the function doc. This is not allowed. The func_doc_language_specific_pre_processing is not model-specific (on the other hand, functions like convert_to_tool are model-specific); we perform it for all models regardless, you can think of it as part of the dataset transformation process (before we enter the model-specific format transformation phase)

[XuHwang]

Compared with the current latest LlamaHandler from main branch commit eca516a, your ToolACEHandler differs in the following two ways:

1. Your _format_prompt has two more \n after each <|end_header_id|> tag. This is an oversight on our end; we missed it for the Llama models. I have just fixed it in [BFCL] Fix Llama Handler #626. After the fix, they should be the exact same.
2. You choose to add the language_specific_hint to the end of the last user prompt instead of as part of the function description in the function doc. This is not allowed. The func_doc_language_specific_pre_processing is not model-specific (on the other hand, functions like convert_to_tool are model-specific); we perform it for all models regardless, you can think of it as part of the dataset transformation process (before we enter the model-specific format transformation phase)

Thanks for the explanation and sorry for the late reply and updates.
We follow your suggestions for using LlamaHandler as our model. And our model is updated, maybe you need to update the local copy if you have always downloaded it before.

[XuHwang]

We close this PR because we have updated the ToolACE handler to make it compatible with the latest BFCL in #653 .