fix extra_match low if batch_size > 1

[sywangyi]

No description provided.

[sywangyi]

lm_eval --model local-completions --tasks gsm8k --batch_size 1 --seed 10 --limit 20 --model_args model=meta-llama/Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:80/v1/completions,num_concurrent=4,tokenized_requests=False,do_sample=False,seed=10

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.7	±	0.1051
		strict-match	5	exact_match	↑	0.6	±	0.1124

before fix :
lm_eval --model local-completions --tasks gsm8k --batch_size 2 --seed 10 --limit 20 --model_args model=meta-llama/Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:80/v1/completions,num_concurrent=4,tokenized_requests=False,do_sample=False,seed=10

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.30	±	0.1051
		strict-match	5	exact_match	↑	0.25	±	0.0993
batch_size =2 and exact_match drop

after fix:
lm_eval --model local-completions --tasks gsm8k --batch_size 2 --seed 10 --limit 20 --model_args model=meta-llama/Llama-3.1-8B-Instruct,base_url=http://127.0.0.1:80/v1/completions,num_concurrent=4,tokenized_requests=False,do_sample=False,seed=10

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.7	±	0.1051
		strict-match	5	exact_match	↑	0.6	±	0.1124

using same tgi server.

[sywangyi]

https://platform.openai.com/docs/api-reference/completions/object. see this. there's a 'index' in "choices" array to indicate the index in batch

[baberabb]

Was the bug that the choices were appearing out out order?

[sywangyi]

Was the bug that the choices were appearing out out order?

yes, the array items in "choices" do not have "index" in ascend order, the "index" is out of order.

[baberabb]

Thanks! Nice catch!