• Anserini v0.37.0
• Anserini v0.36.1
• Anserini v0.36.0
• Anserini v0.35.1
• Anserini v0.35.0

If you are using Windows, please use WSL2 to build Anserini. Please refer to the WSL2 Installation document to install WSL2 if you haven't already.

Note that on Windows without WSL2, tests may fail due to encoding issues, see #1466. A simple workaround is to skip tests by adding -Dmaven.test.skip=true to the above mvn command. See #1121 for additional discussions on debugging Windows build errors.

Key:

• 🔑 = keyword queries
• "full" = full 32-bit floating precision
• "int8" = quantized 8-bit precision
• 🫙 = cached queries, 🅾️ = query encoding with ONNX

The MS MARCO V2.1 corpora were derived from the V2 corpora for the TREC 2024 RAG Track. The experiments below capture topics and qrels originally targeted at the V2 corpora, but have been "projected" over to the V2.1 corpora.

Key:

• F1 = "flat" baseline (Lucene analyzer), keyword queries (🔑)
• F2 = "flat" baseline (pre-tokenized with bert-base-uncased tokenizer), keyword queries (🔑)
• MF = "multifield" baseline (Lucene analyzer), keyword queries (🔑)
• U1 = uniCOIL (noexp), cached queries (🫙)
• S1 = SPLADE++ CoCondenser-EnsembleDistil: cached queries (🫙), ONNX (🅾️)
• BGE (flat) = BGE-base-en-v1.5 (flat indexes)
  • original (float32) indexes: cached queries (🫙), ONNX (🅾️)
  • quantized (int8) indexes: cached queries (🫙), ONNX (🅾️)
• BGE (HNSW) = BGE-base-en-v1.5 (HNSW indexes)
  • original (float32) indexes: cached queries (🫙), ONNX (🅾️)
  • quantized (int8) indexes: cached queries (🫙), ONNX (🅾️)

See instructions below the table for how to reproduce results for a model on all BEIR corpora "in one go".

To reproduce the SPLADE++ CoCondenser-EnsembleDistil results, start by downloading the collection:

wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-splade-pp-ed.tar -P collections/
tar xvf collections/beir-v1.0.0-splade-pp-ed.tar -C collections/

The tarball is 42 GB and has MD5 checksum 9c7de5b444a788c9e74c340bf833173b. Once you've unpacked the data, the following commands will loop over all BEIR corpora and run the regressions:

MODEL="splade-pp-ed"; CORPORA=(trec-covid bioasq nfcorpus nq hotpotqa fiqa signal1m trec-news robust04 arguana webis-touche2020 cqadupstack-android cqadupstack-english cqadupstack-gaming cqadupstack-gis cqadupstack-mathematica cqadupstack-physics cqadupstack-programmers cqadupstack-stats cqadupstack-tex cqadupstack-unix cqadupstack-webmasters cqadupstack-wordpress quora dbpedia-entity scidocs fever climate-fever scifact); for c in "${CORPORA[@]}"
do
    echo "Running $c..."
    python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-${c}.${MODEL}.onnx > logs/log.beir-v1.0.0-${c}-${MODEL}.onnx 2>&1
done

You can verify the results by examining the log files in logs/.

For the other models, modify the above commands as follows:

The "Corpus" above should be substituted into the full file name beir-v1.0.0-${corpus}.tar, e.g., beir-v1.0.0-bge-base-en-v1.5.tar. The above commands should work with some minor modifications: you'll need to tweak the --regression parameter to match the schema of the YAML config files in src/main/resources/regression/.

• Regressions for Mr. TyDi (v1.1) baselines: ar, bn, en, fi, id, ja, ko, ru, sw, te, th
• Regressions for MIRACL (v1.0) baselines: ar, bn, en, es, fa, fi, fr, hi, id, ja, ko, ru, sw, te, th, zh
• Regressions for TREC 2022 NeuCLIR Track BM25 (query translation): Persian, Russian, Chinese
• Regressions for TREC 2022 NeuCLIR Track BM25 (document translation): Persian, Russian, Chinese
• Regressions for TREC 2022 NeuCLIR Track SPLADE (query translation): Persian, Russian, Chinese
• Regressions for TREC 2022 NeuCLIR Track SPLADE (document translation): Persian, Russian, Chinese
• Regressions for HC4 (v1.0) baselines on HC4 corpora: Persian, Russian, Chinese
• Regressions for HC4 (v1.0) baselines on original NeuCLIR22 corpora: Persian, Russian, Chinese
• Regressions for HC4 (v1.0) baselines on translated NeuCLIR22 corpora: Persian, Russian, Chinese
• Regressions for NTCIR-8 ACLIA (IR4QA subtask, Monolingual Chinese)
• Regressions for CLEF 2006 Monolingual French
• Regressions for TREC 2002 Monolingual Arabic
• Regressions for FIRE 2012 monolingual baselines: Bengali, Hindi, English
• Regressions for CIRAL (v1.0) BM25 (query translation): Hausa, Somali, Swahili, Yoruba
• Regressions for CIRAL (v1.0) BM25 (document translation): Hausa, Somali, Swahili, Yoruba

• Regressions for Disks 1 & 2 (TREC 1-3), Disks 4 & 5 (TREC 7-8, Robust04), AQUAINT (Robust05)
• Regressions for the New York Times Corpus (Core17), the Washington Post Corpus (Core18)
• Regressions for Wt10g, Gov2
• Regressions for ClueWeb09 (Category B), ClueWeb12-B13, ClueWeb12
• Regressions for Tweets2011 (MB11 & MB12), Tweets2013 (MB13 & MB14)
• Regressions for Complex Answer Retrieval (CAR17): v1.5, v2.0, v2.0 with doc2query
• Regressions for TREC News Tracks (Background Linking Task): 2018, 2019, 2020
• Regressions for FEVER Fact Verification
• Regressions for DPR Wikipedia QA baselines: 100-word splits, 6/3 sliding window sentences

• Reproducing BM25 baselines for MS MARCO Passage Ranking
• Reproducing BM25 baselines for MS MARCO Document Ranking
• Reproducing baselines for the MS MARCO Document Ranking Leaderboard
• Reproducing doc2query results (MS MARCO Passage Ranking and TREC-CAR)
• Reproducing docTTTTTquery results (MS MARCO Passage and Document Ranking)
• Notes about reproduction issues with MS MARCO Document Ranking w/ docTTTTTquery

• Reproducing BM25 baselines on the MS MARCO V2 Collections

• Indexing AI2's COVID-19 Open Research Dataset
• Baselines for the TREC-COVID Challenge
• Baselines for the TREC-COVID Challenge using doc2query

• Working with the 20 Newsgroups Dataset
• Guide to BM25 baselines for the FEVER Fact Verification Task
• Guide to reproducing "Neural Hype" Experiments
• Guide to running experiments on the AI2 Open Research Corpus
• Experiments from Yang et al. (JDIQ 2018)
• Runbooks for TREC 2018: [Anserini group] [h2oloo group]
• Runbook for ECIR 2019 paper on axiomatic semantic term matching
• Runbook for ECIR 2019 paper on cross-collection relevance feedback
• Support for approximate nearest-neighbor search on dense vectors with inverted indexes

• v0.16.2: December 12, 2022 [Release Notes]
• v0.16.1: November 2, 2022 [Release Notes]
• v0.16.0: October 23, 2022 [Release Notes]
• v0.15.0: September 22, 2022 [Release Notes]
• v0.14.4: July 31, 2022 [Release Notes]
• v0.14.3: May 9, 2022 [Release Notes]
• v0.14.2: March 24, 2022 [Release Notes]
• v0.14.1: February 27, 2022 [Release Notes]
• v0.14.0: January 10, 2022 [Release Notes]
• v0.13.5: November 2, 2021 [Release Notes]
• v0.13.4: October 22, 2021 [Release Notes]
• v0.13.3: August 22, 2021 [Release Notes]
• v0.13.2: July 20, 2021 [Release Notes]
• v0.13.1: June 29, 2021 [Release Notes]
• v0.13.0: June 22, 2021 [Release Notes]
• v0.12.0: April 29, 2021 [Release Notes]
• v0.11.0: February 13, 2021 [Release Notes]
• v0.10.1: January 8, 2021 [Release Notes]
• v0.10.0: November 25, 2020 [Release Notes]
• v0.9.4: June 25, 2020 [Release Notes]
• v0.9.3: May 26, 2020 [Release Notes]
• v0.9.2: May 14, 2020 [Release Notes]
• v0.9.1: May 6, 2020 [Release Notes]
• v0.9.0: April 18, 2020 [Release Notes]
• v0.8.1: March 22, 2020 [Release Notes]
• v0.8.0: March 11, 2020 [Release Notes]
• v0.7.2: January 25, 2020 [Release Notes]
• v0.7.1: January 9, 2020 [Release Notes]
• v0.7.0: December 13, 2019 [Release Notes]
• v0.6.0: September 6, 2019 [Release Notes] [Known Issues]
• v0.5.1: June 11, 2019 [Release Notes]
• v0.5.0: June 5, 2019 [Release Notes]
• v0.4.0: March 4, 2019 [Release Notes]
• v0.3.0: December 16, 2018 [Release Notes]
• v0.2.0: September 10, 2018 [Release Notes]
• v0.1.0: July 4, 2018 [Release Notes]

• Anserini was upgraded to Lucene 9.3 at commit 272565 (8/2/2022): this upgrade created backward compatibility issues, see #1952. Anserini will automatically detect Lucene 8 indexes and disable consistent tie-breaking to avoid runtime errors. However, Lucene 9 code running on Lucene 8 indexes may give slightly different results than Lucene 8 code running on Lucene 8 indexes. Lucene 8 code will not run on Lucene 9 indexes. Pyserini has also been upgraded and similar issues apply: Lucene 9 code running on Lucene 8 indexes may give slightly different results than Lucene 8 code running on Lucene 8 indexes.
• Anserini was upgraded to Java 11 at commit 17b702d (7/11/2019) from Java 8. Maven 3.3+ is also required.
• Anserini was upgraded to Lucene 8.0 as of commit 75e36f9 (6/12/2019); prior to that, the toolkit uses Lucene 7.6. Based on preliminary experiments, query evaluation latency has been much improved in Lucene 8. As a result of this upgrade, results of all regressions have changed slightly. To reproducible old results from Lucene 7.6, use v0.5.1.