Anserini: Regressions for the Washington Post (Core18)
This page describes regressions for the TREC 2018 Common Core Track, which uses the TREC Washington Post Corpus. The exact configurations for these regressions are stored in this YAML file. Note that this page is automatically generated from this template as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.
Typical indexing command:
nohup sh target/appassembler/bin/IndexCollection -collection WashingtonPostCollection -input /path/to/core18 \
-index lucene-index.core18.pos+docvectors+rawdocs -generator WashingtonPostGenerator -threads 1 \
-storePositions -storeDocvectors -storeRaw >& log.core18.pos+docvectors+rawdocs &
The directory /path/to/core18/
should be the root directory of the TREC Washington Post Corpus, i.e., ls /path/to/core18/
should bring up a single JSON file.
For additional details, see explanation of common indexing options.
Topics and qrels are stored in src/main/resources/topics-and-qrels/
, downloaded from NIST:
topics.core18.txt
: topics for the TREC 2018 Common Core Trackqrels.core18.txt
: qrels for the TREC 2018 Common Core Track
After indexing has completed, you should be able to perform retrieval as follows:
nohup target/appassembler/bin/SearchCollection -index lucene-index.core18.pos+docvectors+rawdocs \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.core18.txt \
-bm25 -output run.core18.bm25.topics.core18.txt &
nohup target/appassembler/bin/SearchCollection -index lucene-index.core18.pos+docvectors+rawdocs \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.core18.txt \
-bm25 -rm3 -output run.core18.bm25+rm3.topics.core18.txt &
nohup target/appassembler/bin/SearchCollection -index lucene-index.core18.pos+docvectors+rawdocs \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.core18.txt \
-bm25 -axiom -axiom.deterministic -rerankCutoff 20 -output run.core18.bm25+ax.topics.core18.txt &
nohup target/appassembler/bin/SearchCollection -index lucene-index.core18.pos+docvectors+rawdocs \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.core18.txt \
-qld -output run.core18.ql.topics.core18.txt &
nohup target/appassembler/bin/SearchCollection -index lucene-index.core18.pos+docvectors+rawdocs \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.core18.txt \
-qld -rm3 -output run.core18.ql+rm3.topics.core18.txt &
nohup target/appassembler/bin/SearchCollection -index lucene-index.core18.pos+docvectors+rawdocs \
-topicreader Trec -topics src/main/resources/topics-and-qrels/topics.core18.txt \
-qld -axiom -axiom.deterministic -rerankCutoff 20 -output run.core18.ql+ax.topics.core18.txt &
Evaluation can be performed using trec_eval
:
eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.core18.txt run.core18.bm25.topics.core18.txt
eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.core18.txt run.core18.bm25+rm3.topics.core18.txt
eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.core18.txt run.core18.bm25+ax.topics.core18.txt
eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.core18.txt run.core18.ql.topics.core18.txt
eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.core18.txt run.core18.ql+rm3.topics.core18.txt
eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.core18.txt run.core18.ql+ax.topics.core18.txt
With the above commands, you should be able to replicate the following results:
MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
---|---|---|---|---|---|---|
TREC 2018 Common Core Track Topics | 0.2495 | 0.3135 | 0.2841 | 0.2526 | 0.3073 | 0.2919 |
P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
---|---|---|---|---|---|---|
TREC 2018 Common Core Track Topics | 0.3567 | 0.4200 | 0.3947 | 0.3653 | 0.4000 | 0.4020 |
- Results replicated by @andrewyates on 2018-11-30 (commit
c1aac5
) - Results replicated by @chriskamphuis on 2019-09-07 (commit
61f6f20
) - Results replicated by @nikhilro on 2020-01-26 (commit
d5ee069
) - Results replicated by @edwinzhng on 2020-01-26 (commit
7b76dfb
)