Skip to content

Commit

Permalink
Added Hellaswag
Browse files Browse the repository at this point in the history
  • Loading branch information
Kirill Semin committed Feb 5, 2024
1 parent 29e9ed5 commit e975030
Show file tree
Hide file tree
Showing 4 changed files with 5 additions and 1 deletion.
1 change: 1 addition & 0 deletions lm_eval/tasks/arc/arc_easy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ doc_to_target: "{{choices.label.index(answerKey)}}"
doc_to_choice: "{{choices.text}}"
should_decontaminate: true
doc_to_decontamination_query: "Question: {{question}}\nAnswer:"
num_fewshot: 25
metric_list:
- metric: acc
aggregation: mean
Expand Down
2 changes: 2 additions & 0 deletions lm_eval/tasks/arc/arc_easy_ru.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@ include: arc_easy.yaml
task: arc_easy_ru
dataset_path: deepvk/ARC-easy-ru
dataset_name: null
doc_to_text: "Вопрос: {{question}}\nОтвет:"
doc_to_decontamination_query: "Вопрос: {{question}}\nОтвет:"
2 changes: 1 addition & 1 deletion lm_eval/tasks/gsm8k/gsm8k-ru.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
include: gsm8k.yaml
group:
- math_word_problems
task: gsm8k-ru
task: gsm8k_ru
dataset_path: deepvk/gsm8k-ru
dataset_name: null
doc_to_text: "Вопрос: {{question}}\nОтвет:"
Expand Down
1 change: 1 addition & 0 deletions lm_eval/tasks/hellaswag/hellaswag.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ output_type: multiple_choice
training_split: train
validation_split: validation
test_split: null
num_fewshot: 10
process_docs: !function utils.process_docs
doc_to_text: "{{query}}"
doc_to_target: "{{label}}"
Expand Down

0 comments on commit e975030

Please sign in to comment.