From 9fd3f6cdf90e01752d893b43a7431c16deec61fd Mon Sep 17 00:00:00 2001 From: Lj Miranda <12949683+ljvmiranda921@users.noreply.github.com> Date: Tue, 27 Aug 2024 10:21:06 -0700 Subject: [PATCH] Update docs (#29) --- README.md | 130 +++++++----------------------------------------------- docs.md | 109 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+), 113 deletions(-) create mode 100644 docs.md diff --git a/README.md b/README.md index f5f0f4e..0018663 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,22 @@ -# Aya Expedition: Reward Model Multilingual +# Expedition Aya: Reward Model Multilingual -Repository for Aya Expedition Project : Reward Model Multilingual +In this project, we evaluate reward models on their multilingual capabilities. +We translated [RewardBench](https://huggingface.co/datasets/allenai/reward-bench) into 23 diverse languages and evaluated several open-source and multilingual LLMs on their chat, safety, and reasoning capabilities. +This project is part of [Cohere for AI's Expedition Aya 2024](https://sites.google.com/cohere.com/expedition-aya/home), a 6-week open build challenge. -Project Docs: [docs](https://docs.google.com/document/d/11l7Mb60JMRpdJpp9-B7VjWOF4FshBdjzY0FDOTq9sMk/edit?usp=sharing) +

+🤗 Dataset +| +💬 Presentation +| +📚 Documentation +| +📄 Paper (coming soon!) +

+ +## Team Members + +The team is composed of Srishti Gureja (@srishti-git1110), Shayekh Bin Islam, (@ShayekhBinIslam), Rishabh Maheshwary (@RishabhMaheshwary), Drishti Sushma (@DrishtiShrrrma), Gusti Winata (@sanggusti), and Lj Miranda (@ljvmiranda921). ## Setup and installation @@ -18,116 +32,6 @@ pip install -r requirements.txt Note that the [`rewardbench`](https://pypi.org/project/rewardbench/) package requires Python 3.10 and above. -## Running experiments - -First, you need to set a [HuggingFace token](https://huggingface.co/settings/tokens) as an environment variable (`HF_TOKEN`): - -```sh -export HF_TOKEN= -``` - -You can find all runnable experiments in the `scripts` directory. -Their filename should explicitly tell you their purpose. - -### Running translation - -We currently use [`facebook/nllb-200-3.3B`](https://huggingface.co/facebook/nllb-200-3.3B) for translation. First install sentence splitter using: - -``` -pip install git+https://github.com/mediacloud/sentence-splitter.git -``` - -To translate reward bench into [22 Aya languages](https://arxiv.org/abs/2405.15032) run the following: - -``` -cd scripts -bash run_nllb.sh -``` - -You can also translate a specifc preference dataset from huggingface to a specifc target language using `scripts/translate_preference_pairs_nllb.py`. - -### Getting rewards from a Reward Model (RM) on a HuggingFace dataset - -Here, we use the `scripts/run_rewardbench.py` command-line interface and pass a HuggingFace dataset. -This is useful if the reward model is trained as a Custom classifier (🛠️), Sequence classifier (🔢), or via DPO (🎯). -For example, if we want to get the reward score of the UltraRM-13b reward model on a preference dataset, we run: - -```sh -python -m scripts.run_rewardbench \ - --model openbmb/UltraRM-13b \ - --chat_template openbmb \ - --dataset_name $DATASET \ - --lang_code $LANG_CODE \ - --split "filtered" \ - --output_dir $OUTDIR \ - --batch_size 8 \ - --trust_remote_code \ - --force_truncation \ - --save_all -``` - -The evaluation parameters can be found in the [allenai/reward-bench](https://github.com/allenai/reward-bench/blob/main/scripts/configs/eval_configs.yaml) repository. -This runs the reward model on the (prompt, chosen, rejected) triples and give us the reward score for each instance. -The results are saved into a JSON file inside the `$OUTDIR` directory. -Finally, you can find some experiments in the `experiments/run_rm_evals.sh` script. - -### Getting rewards from a Generative RM on a HuggingFace dataset - -Here we use `scripts/run_generative.py`, a modified version of the [same script in RewardBench](https://github.com/allenai/reward-bench/blob/main/scripts/run_generative.py) to obtain rewards from a Generative RM (🗨️). -The only difference is that this script accepts any arbitrary HuggingFace preference dataset (which we plan to conribute upstream later on) instead of just the RewardBench dataset. - -For Generative RMs, we prompt a model in a style akin to LLM-as-a-judge, and then parse the output to obtain the preference. -This can be done for closed-source APIs (e.g., GPT-4, Claude) or open-source LMs (done via vLLM). -If you're planning to use some closed-source APIs, you also need to set the tokens for each: - -```sh -export OPENAI_API_KEY= -export CO_API_KEY= -export ANTHROPIC_API_KEY= -``` - -**You can also store all your API keys in a .env file.** -It will be loaded using the [python-dotenv library](https://github.com/theskumar/python-dotenv). -Say we want to obtain the preferences of `gpt-4-2024-04-09`: - -```sh -export OPENAI_API_KEY= -python -m scripts.run_generative \ - --dataset_name $DATASET \ - --model gpt-4-turbo-2024-04-09 \ - --split "filtered" \ - --lang_code $LANG_CODE \ - --output_dir $OUTDIR -``` - -You can also run open-source LMs in a generative fashion. -The inference is then routed through [vLLM](https://github.com/vllm-project/vllm). -Here's an example using `meta-llama/Meta-Llama-3-70B-Instruct`: - -```sh -python -m scripts/run_generative.py \ - --dataset_name $DATASET \ - --lang_code $LANG_CODE \ - --split "filtered" \ - --model "meta-llama/Meta-Llama-3-70B-Instruct" \ - --num_gpus 4 \ - --output_dir $OUTDIR -``` - -To improve prompt output especially on multilingual cases, we recommend passing a tuple to the `--include_languages` parameter. -The first value should be the language a prompt was written in, and the second value should be the language the assistant should use in its answer. - -```diff -python -m scripts/run_generative.py \ - --dataset_name $DATASET \ - --lang_code deu_Latn \ - --split $SPLIT \ - --model "meta-llama/Meta-Llama-3-70B-Instruct" \ - --num_gpus 4 \ -+ --include_languages German English - --output_dir $OUTDIR -``` - ## Testing and Development This codebase contains minimal tests, mostly we test functions that were added or patched from RewardBench. diff --git a/docs.md b/docs.md new file mode 100644 index 0000000..de1fbe0 --- /dev/null +++ b/docs.md @@ -0,0 +1,109 @@ +# Running experiments + +First, you need to set a [HuggingFace token](https://huggingface.co/settings/tokens) as an environment variable (`HF_TOKEN`): + +```sh +export HF_TOKEN= +``` + +You can find all runnable experiments in the `scripts` directory. +Their filename should explicitly tell you their purpose. + +## Running translation + +We currently use [`facebook/nllb-200-3.3B`](https://huggingface.co/facebook/nllb-200-3.3B) for translation. First install sentence splitter using: + +``` +pip install git+https://github.com/mediacloud/sentence-splitter.git +``` + +To translate reward bench into [22 Aya languages](https://arxiv.org/abs/2405.15032) run the following: + +``` +cd scripts +bash run_nllb.sh +``` + +You can also translate a specifc preference dataset from huggingface to a specifc target language using `scripts/translate_preference_pairs_nllb.py`. + +## Getting rewards from a Reward Model (RM) on a HuggingFace dataset + +Here, we use the `scripts/run_rewardbench.py` command-line interface and pass a HuggingFace dataset. +This is useful if the reward model is trained as a Custom classifier (🛠️), Sequence classifier (🔢), or via DPO (🎯). +For example, if we want to get the reward score of the UltraRM-13b reward model on a preference dataset, we run: + +```sh +python -m scripts.run_rewardbench \ + --model openbmb/UltraRM-13b \ + --chat_template openbmb \ + --dataset_name $DATASET \ + --lang_code $LANG_CODE \ + --split "filtered" \ + --output_dir $OUTDIR \ + --batch_size 8 \ + --trust_remote_code \ + --force_truncation \ + --save_all +``` + +The evaluation parameters can be found in the [allenai/reward-bench](https://github.com/allenai/reward-bench/blob/main/scripts/configs/eval_configs.yaml) repository. +This runs the reward model on the (prompt, chosen, rejected) triples and give us the reward score for each instance. +The results are saved into a JSON file inside the `$OUTDIR` directory. +Finally, you can find some experiments in the `experiments/run_rm_evals.sh` script. + +## Getting rewards from a Generative RM on a HuggingFace dataset + +Here we use `scripts/run_generative.py`, a modified version of the [same script in RewardBench](https://github.com/allenai/reward-bench/blob/main/scripts/run_generative.py) to obtain rewards from a Generative RM (🗨️). +The only difference is that this script accepts any arbitrary HuggingFace preference dataset (which we plan to conribute upstream later on) instead of just the RewardBench dataset. + +For Generative RMs, we prompt a model in a style akin to LLM-as-a-judge, and then parse the output to obtain the preference. +This can be done for closed-source APIs (e.g., GPT-4, Claude) or open-source LMs (done via vLLM). +If you're planning to use some closed-source APIs, you also need to set the tokens for each: + +```sh +export OPENAI_API_KEY= +export CO_API_KEY= +export ANTHROPIC_API_KEY= +``` + +**You can also store all your API keys in a .env file.** +It will be loaded using the [python-dotenv library](https://github.com/theskumar/python-dotenv). +Say we want to obtain the preferences of `gpt-4-2024-04-09`: + +```sh +export OPENAI_API_KEY= +python -m scripts.run_generative \ + --dataset_name $DATASET \ + --model gpt-4-turbo-2024-04-09 \ + --split "filtered" \ + --lang_code $LANG_CODE \ + --output_dir $OUTDIR +``` + +You can also run open-source LMs in a generative fashion. +The inference is then routed through [vLLM](https://github.com/vllm-project/vllm). +Here's an example using `meta-llama/Meta-Llama-3-70B-Instruct`: + +```sh +python -m scripts/run_generative.py \ + --dataset_name $DATASET \ + --lang_code $LANG_CODE \ + --split "filtered" \ + --model "meta-llama/Meta-Llama-3-70B-Instruct" \ + --num_gpus 4 \ + --output_dir $OUTDIR +``` + +To improve prompt output especially on multilingual cases, we recommend passing a tuple to the `--include_languages` parameter. +The first value should be the language a prompt was written in, and the second value should be the language the assistant should use in its answer. + +```diff +python -m scripts/run_generative.py \ + --dataset_name $DATASET \ + --lang_code deu_Latn \ + --split $SPLIT \ + --model "meta-llama/Meta-Llama-3-70B-Instruct" \ + --num_gpus 4 \ ++ --include_languages German English + --output_dir $OUTDIR +```