Skip to content

Latest commit

 

History

History
469 lines (384 loc) · 70.1 KB

README.md

File metadata and controls

469 lines (384 loc) · 70.1 KB
Raw

Anserini

build codecov Generic badge Maven Central LICENSE doi

Anserini is a toolkit for reproducible information retrieval research. By building on Lucene, we aim to bridge the gap between academic information retrieval research and the practice of building real-world search applications. Among other goals, our effort aims to be the opposite of this.* Anserini grew out of a reproducibility study of various open-source retrieval engines in 2016 (Lin et al., ECIR 2016). See Yang et al. (SIGIR 2017) and Yang et al. (JDIQ 2018) for overviews.

🎬 Getting Started

Most Anserini features are exposed in the Pyserini Python interface. If you're more comfortable with Python, start there, although Anserini forms an important building block of Pyserini, so it remains worthwhile to learn about Anserini.

You'll need Java 11 and Maven 3.3+ to build Anserini. Clone our repo with the --recurse-submodules option to make sure the eval/ submodule also gets cloned (alternatively, use git submodule update --init). Then, build using using Maven:

mvn clean package appassembler:assemble

The tools/ directory, which contains evaluation tools and other scripts, is actually this repo, integrated as a Git submodule (so that it can be shared across related projects). Build as follows (you might get warnings, but okay to ignore):

cd tools/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make && cd ../../..
cd tools/eval/ndeval && make && cd ../../..

With that, you should be ready to go. The onboarding path for Anserini starts here!

Windows tips

If you are using Windows, please use WSL2 to build Anserini. Please refer to the WSL2 Installation document to install WSL2 if you haven't already.

Note that on Windows without WSL2, tests may fail due to encoding issues, see #1466. A simple workaround is to skip tests by adding -Dmaven.test.skip=true to the above mvn command. See #1121 for additional discussions on debugging Windows build errors.

⚗️ Regression Experiments (+ Reproduction Guides)

Anserini is designed to support experiments on various standard IR test collections out of the box. The following experiments are backed by rigorous end-to-end regression tests with run_regression.py and the Anserini reproducibility promise. For the most part, these runs are based on default parameter settings. These pages can also serve as guides to reproduce our results. See individual pages for details!

MS MARCO V1 Passage Regressions

MS MARCO V1 Passage Regressions

dev DL19 DL20
Unsupervised Sparse
BoW baselines + + +
Quantized BM25 ✓ ✓ ✓
WP baselines + + +
Huggingface WP baselines + + +
doc2query +
doc2query-T5 + + +
Learned Sparse (uniCOIL family)
uniCOIL noexp ✓ ✓ ✓
uniCOIL with doc2query-T5 ✓ ✓ ✓
uniCOIL with TILDE ✓
Learned Sparse (other)
DeepImpact ✓
SPLADEv2 ✓
SPLADE++ CoCondenser-EnsembleDistil ✓ ✓ ✓
SPLADE++ CoCondenser-EnsembleDistil (ONNX) ✓ ✓ ✓
SPLADE++ CoCondenser-SelfDistil ✓ ✓ ✓
SPLADE++ CoCondenser-SelfDistil (ONNX) ✓ ✓ ✓
Learned Dense (HNSW)
cosDPR-distil w/ HNSW fp32 ✓ ✓ ✓
cosDPR-distil w/ HSNW fp32 (ONNX) ✓ ✓ ✓
cosDPR-distil w/ HNSW int8 ✓ ✓ ✓
cosDPR-distil w/ HSNW int8 (ONNX) ✓ ✓ ✓
BGE-base-en-v1.5 w/ HNSW fp32 ✓ ✓ ✓
BGE-base-en-v1.5 w/ HNSW fp32 (ONNX) ✓ ✓ ✓
BGE-base-en-v1.5 w/ HNSW int8 ✓ ✓ ✓
BGE-base-en-v1.5 w/ HNSW int8 (ONNX) ✓ ✓ ✓
OpenAI Ada2 w/ HNSW fp32 ✓ ✓ ✓
OpenAI Ada2 w/ HNSW int8 ✓ ✓ ✓
Cohere English v3 w/ HNSW fp32 ✓
Cohere English v3 w/ HNSW int8 ✓
Learned Dense (Inverted; experimental)
cosDPR-distil w/ "fake words" ✓ ✓ ✓
cosDPR-distil w/ "LexLSH" ✓ ✓ ✓

Available Corpora for Download

Corpora Size Checksum
Quantized BM25 1.2 GB 0a623e2c97ac6b7e814bf1323a97b435
uniCOIL (noexp) 2.7 GB f17ddd8c7c00ff121c3c3b147d2e17d8
uniCOIL (d2q-T5) 3.4 GB 78eef752c78c8691f7d61600ceed306f
uniCOIL (TILDE) 3.9 GB 12a9c289d94e32fd63a7d39c9677d75c
DeepImpact 3.6 GB 73843885b503af3c8b3ee62e5f5a9900
SPLADEv2 9.9 GB b5d126f5d9a8e1b3ef3f5cb0ba651725
SPLADE++ CoCondenser-EnsembleDistil 4.2 GB e489133bdc54ee1e7c62a32aa582bc77
SPLADE++ CoCondenser-SelfDistil 4.8 GB cb7e264222f2bf2221dd2c9d28190be1
cosDPR-distil 57 GB e20ffbc8b5e7f760af31298aefeaebbd
BGE-base-en-v1.5 59 GB 353d2c9e72e858897ad479cca4ea0db1
OpenAI-ada2 109 GB a4d843d522ff3a3af7edbee789a63402
Cohere embed-english-v3.0 38 GB 6b7d9795806891b227378f6c290464a9
MS MARCO V1 Document Regressions

MS MARCO V1 Document Regressions

dev DL19 DL20
Unsupervised Lexical, Complete Doc*
BoW baselines + + +
WP baselines + + +
Huggingface WP baselines + + +
doc2query-T5 + + +
Unsupervised Lexical, Segmented Doc*
BoW baselines + + +
WP baselines + + +
doc2query-T5 + + +
Learned Sparse Lexical
uniCOIL noexp ✓ ✓ ✓
uniCOIL with doc2query-T5 ✓ ✓ ✓

Available Corpora for Download

Corpora Size Checksum
MS MARCO V1 doc: uniCOIL (noexp) 11 GB 11b226e1cacd9c8ae0a660fd14cdd710
MS MARCO V1 doc: uniCOIL (d2q-T5) 19 GB 6a00e2c0c375cb1e52c83ae5ac377ebb
MS MARCO V2 Passage Regressions

MS MARCO V2 Passage Regressions

dev DL21 DL22
Unsupervised Lexical, Original Corpus
baselines + + +
doc2query-T5 + + +
Unsupervised Lexical, Augmented Corpus
baselines + + +
doc2query-T5 + + +
Learned Sparse Lexical
uniCOIL noexp zero-shot ✓ ✓ ✓
uniCOIL with doc2query-T5 zero-shot ✓ ✓ ✓
SPLADE++ CoCondenser-EnsembleDistil ✓ ✓ ✓
SPLADE++ CoCondenser-SelfDistil ✓ ✓ ✓

Available Corpora for Download

Corpora Size Checksum
uniCOIL (noexp) 24 GB d9cc1ed3049746e68a2c91bf90e5212d
uniCOIL (d2q-T5) 41 GB 1949a00bfd5e1f1a230a04bbc1f01539
SPLADE++ CoCondenser-EnsembleDistil 66 GB 2cdb2adc259b8fa6caf666b20ebdc0e8
SPLADE++ CoCondenser-SelfDistil) 76 GB 061930dd615c7c807323ea7fc7957877
MS MARCO V2 Document Regressions

MS MARCO V2 Document Regressions

dev DL21
Unsupervised Lexical, Complete Doc
baselines + +
doc2query-T5 + +
Unsupervised Lexical, Segmented Doc
baselines + +
doc2query-T5 + +
Learned Sparse Lexical
uniCOIL noexp zero-shot ✓ ✓
uniCOIL with doc2query-T5 zero-shot ✓ ✓

Available Corpora for Download

Corpora Size Checksum
MS MARCO V2 doc: uniCOIL (noexp) 55 GB 97ba262c497164de1054f357caea0c63
MS MARCO V2 doc: uniCOIL (d2q-T5) 72 GB c5639748c2cbad0152e10b0ebde3b804
BEIR (v1.0.0) Regressions

BEIR (v1.0.0) Regressions

Key:

  • F1 = "flat" baseline (Lucene analyzer)
  • F2 = "flat" baseline (pre-tokenized with bert-base-uncased tokenizer)
  • MF = "multifield" baseline (Lucene analyzer)
  • U1 = uniCOIL (noexp)
  • S1 = SPLADE++ CoCondenser-EnsembleDistil: pre-encoded queries (✓), ONNX (O)
  • D1 = BGE-base-en-v1.5: original HNSW indexes
  • D2 = BGE-base-en-v1.5: quantized HNSW indexes

See instructions below the table for how to reproduce results for a model on all BEIR corpora "in one go".

Corpus F1 F2 MF U1 S1 D1 D2
TREC-COVID ✓ ✓ ✓ ✓ ✓ O ✓ ✓
BioASQ ✓ ✓ ✓ ✓ ✓ O ✓ ✓
NFCorpus ✓ ✓ ✓ ✓ ✓ O ✓ ✓
NQ ✓ ✓ ✓ ✓ ✓ O ✓ ✓
HotpotQA ✓ ✓ ✓ ✓ ✓ O ✓ ✓
FiQA-2018 ✓ ✓ ✓ ✓ ✓ O ✓ ✓
Signal-1M(RT) ✓ ✓ ✓ ✓ ✓ O ✓ ✓
TREC-NEWS ✓ ✓ ✓ ✓ ✓ O ✓ ✓
Robust04 ✓ ✓ ✓ ✓ ✓ O ✓ ✓
ArguAna ✓ ✓ ✓ ✓ ✓ O ✓ ✓
Touche2020 ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Android ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-English ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Gaming ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Gis ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Mathematica ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Physics ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Programmers ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Stats ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Tex ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Unix ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Webmasters ✓ ✓ ✓ ✓ ✓ O ✓ ✓
CQADupStack-Wordpress ✓ ✓ ✓ ✓ ✓ O ✓ ✓
Quora ✓ ✓ ✓ ✓ ✓ O ✓ ✓
DBPedia ✓ ✓ ✓ ✓ ✓ O ✓ ✓
SCIDOCS ✓ ✓ ✓ ✓ ✓ O ✓ ✓
FEVER ✓ ✓ ✓ ✓ ✓ O ✓ ✓
Climate-FEVER ✓ ✓ ✓ ✓ ✓ O ✓ ✓
SciFact ✓ ✓ ✓ ✓ ✓ O ✓ ✓

To reproduce the SPLADE++ CoCondenser-EnsembleDistil results, start by downloading the collection:

wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-splade-pp-ed.tar -P collections/
tar xvf collections/beir-v1.0.0-splade-pp-ed.tar -C collections/

The tarball is 42 GB and has MD5 checksum 9c7de5b444a788c9e74c340bf833173b. Once you've unpacked the data, the following commands will loop over all BEIR corpora and run the regressions:

MODEL="splade-pp-ed"; CORPORA=(trec-covid bioasq nfcorpus nq hotpotqa fiqa signal1m trec-news robust04 arguana webis-touche2020 cqadupstack-android cqadupstack-english cqadupstack-gaming cqadupstack-gis cqadupstack-mathematica cqadupstack-physics cqadupstack-programmers cqadupstack-stats cqadupstack-tex cqadupstack-unix cqadupstack-webmasters cqadupstack-wordpress quora dbpedia-entity scidocs fever climate-fever scifact); for c in "${CORPORA[@]}"
do
    echo "Running $c..."
    python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-${c}-${MODEL} > logs/log.beir-v1.0.0-${c}-${MODEL} 2>&1
done

You can verify the results by examining the log files in logs/.

For the other models, modify the above commands as follows:

Key Corpus Checksum MODEL
F1 corpus faefd5281b662c72ce03d22021e4ff6b flat
F2 corpus-wp 3cf8f3dcdcadd49362965dd4466e6ff2 flat-wp
MF corpus faefd5281b662c72ce03d22021e4ff6b multifield
U1 unicoil-noexp 4fd04d2af816a6637fc12922cccc8a83 unicoil-noexp
S1 splade-pp-ed 9c7de5b444a788c9e74c340bf833173b splade-pp-ed
D1 bge-base-en-v1.5 e4e8324ba3da3b46e715297407a24f00 bge-base-en-v1.5-hnsw

The "Corpus" above should be substituted into the full file name beir-v1.0.0-${corpus}.tar, e.g., beir-v1.0.0-bge-base-en-v1.5.tar.

Cross-lingual and Multi-lingual Regressions

Cross-lingual and Multi-lingual Regressions

Other Regressions

Other Regressions

📃 Additional Documentation

The experiments described below are not associated with rigorous end-to-end regression testing and thus provide a lower standard of reproducibility. For the most part, manual copying and pasting of commands into a shell is required to reproduce our results.

MS MARCO V1

MS MARCO V1

MS MARCO V2

MS MARCO V2

TREC-COVID and CORD-19

TREC-COVID and CORD-19

Other Experiments and Features

Other Experiments and Features

🙋 How Can I Contribute?

If you've found Anserini to be helpful, we have a simple request for you to contribute back. In the course of reproducing baseline results on standard test collections, please let us know if you're successful by sending us a pull request with a simple note, like what appears at the bottom of the page for Disks 4 & 5. Reproducibility is important to us, and we'd like to know about successes as well as failures. Since the regression documentation is auto-generated, pull requests should be sent against the raw templates. Then the regression documentation can be generated using the bin/build.sh script. In turn, you'll be recognized as a contributor.

Beyond that, there are always open issues we would appreciate help on!

📜️ Release History

older... (and historic notes)

📜️ Historical Notes

  • Anserini was upgraded to Lucene 9.3 at commit 272565 (8/2/2022): this upgrade created backward compatibility issues, see #1952. Anserini will automatically detect Lucene 8 indexes and disable consistent tie-breaking to avoid runtime errors. However, Lucene 9 code running on Lucene 8 indexes may give slightly different results than Lucene 8 code running on Lucene 8 indexes. Lucene 8 code will not run on Lucene 9 indexes. Pyserini has also been upgraded and similar issues apply: Lucene 9 code running on Lucene 8 indexes may give slightly different results than Lucene 8 code running on Lucene 8 indexes.
  • Anserini was upgraded to Java 11 at commit 17b702d (7/11/2019) from Java 8. Maven 3.3+ is also required.
  • Anserini was upgraded to Lucene 8.0 as of commit 75e36f9 (6/12/2019); prior to that, the toolkit uses Lucene 7.6. Based on preliminary experiments, query evaluation latency has been much improved in Lucene 8. As a result of this upgrade, results of all regressions have changed slightly. To reproducible old results from Lucene 7.6, use v0.5.1.

✨ References

🙏 Acknowledgments

This research is supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada. Previous support came from the U.S. National Science Foundation under IIS-1423002 and CNS-1405688. Any opinions, findings, and conclusions or recommendations expressed do not necessarily reflect the views of the sponsors.