Skip to content

Latest commit

 

History

History
74 lines (54 loc) · 3.68 KB

decontamination.md

File metadata and controls

74 lines (54 loc) · 3.68 KB
Raw

Decontamination

Usage

Simply add a "--decontamination_ngrams_path" when running main.py. The provided directory should contain the ngram files and info.json produced in "Pile Ngram Generation" further down.

python main.py \
    --model gpt2 \
    --device 0 \
    --tasks sciq \
    --decontamination_ngrams_path path/containing/training/set/ngrams

Background

Downstream evaluations test model generalization, and are less useful when test set data also exists in the training set, referred to as leakage or contamination.

Filtering your training set against the test set is a good first step, however this isn't always possible, as in the case of a new benchmark or one that wasn't considered prior to model training. When training set filtering isn't possible, it is useful to measure the impact of test set leakage by detecting the contaminated test examples and producing a clean version of the benchmark.

The basis for our decontamination procedure can be found in Appendix C of "Language Models are Few-Shot Learners". OpenAI defined a test document as contaminated if any N-gram overlap existed with any training document. They used a range of N values between 8 and 13 depending on dataset, while we just used 13 for simplicity.

Implementation

Contamination detection can be found in lm_eval/decontaminate.py with supporting code in lm_eval/decontamination/.

decontaminate.py does the following:

  1. Build dictionaries of all ngrams and their corresponding evaluation/document ids.
  2. Scan through sorted files containing training set n-grams.
  3. If a match is found, the corresponding evaluation/document combinations are marked as contaminated.

lm_eval/evaluator.py can then produce a clean version of the benchmark by excluding the results of contaminated documents. For each metric, a clean version will be shown in the results with a "decontaminate" suffix.

This is disabled by default for new tasks, to support decontamination on a task override the "should_decontaminate" and "doc_to_decontamination_query" methods. For more details see the task guide.

Pile Ngram Generation

The relevant scripts can be found in scripts/clean_training_data, which also import from lm_eval/decontamination/

  1. git clone https://github.com/EleutherAI/lm-evaluation-harness.git
  2. pip install -r requirements.txt
  3. Download The Pile from The Eye
  4. Place pile files in "pile" directory under "lm-evaluation-harness" (or create a symlink)
  5. Run generate_13_grams.
export PYTHONHASHSEED=0
python -m scripts/clean_training_data/generate_13_grams \
       -dir path/to/working/directory \
       -n 13 \
       -buckets 500

Took approximately 4 days for us. We had the time to wait, but this could be scaled out by doing partial pile scans on multiple instances of this script and merging the relevant buckets. We fixed PYTHONHASHSEED to ensure reproducibility of bucket hashing in case you need to stop and start.

  1. Sort the generated 13-grams.
python -m scripts/clean_training_data/sort_13_gram_buckets \
       -dir path/to/working/directory/output

Took approximately 5 days for us. You could speed this up by spreading the files around to different machines and running the sort script before gathering them together.

  1. Compress the sorted 13 grams files and place them together with info.json.

This step only takes a few hours.

python -m scripts/clean_training_data/compress_and_package \
       -dir path/to/working/directory \
       -output path/to/final/directory \
       -procs 8

Congratulations, the final directory can now be passed to lm-evaulation-harness with the "--decontamination_ngrams_path" argument.