This repository contains the code for the 2017 paper "Social Bias in Elicited Natural Language Inferences" by Rachel Rudinger, Chandler May, and Benjamin Van Durme. Rachel Rudinger and Chandler May contributed to this code, which is released under the two-clause BSD license.
Install dependencies with:
pip install -r requirements.txt
Download and unzip the SNLI data:
wget https://nlp.stanford.edu/projects/snli/snli_1.0.zip
unzip snli_1.0.zip
Compute counts for unigrams and bigrams, across all inference types,
using 7 subprocesses (in addition to the main process), and filtering
out hypothesis words that occur in the premise. Read SNLI pairs from
snli_1.0/snli_1.0_train.jsonl
and write counts to
snli_stats/counts.pkl
:
python snli_cooccur.py \
between-prem-hypo \
--max-ngram 2 \
--num-proc 7 \
--filter-hypo-by-prem \
snli_1.0/snli_1.0_train.jsonl snli_stats/counts.pkl
Run python snli_cooccur.py --help
for more options.
Alternatively, compute counts for all parameter configurations, in a loop:
bash snli_cooccur_loop.bash
To change the default input and output directories, or change the
Python interpreter used to run snli_cooccur.py
, create a file named
snli_cooccur_loop_include.bash
with the following contents and
modify them as desired (and then run snli_cooccur_loop.bash
):
snli_dir=snli_1.0
output_dir=snli_stats
big_python=python
little_python=python
The little_python
and big_python
variables are the Python
commands used for unigram and unigram-and-bigram models, respectively.
The latter have higher memory requirements.
(Note the little_python
and big_python
variables can be set to job
submission scripts invoking a python interpreter to parallelize the
computation on a grid.)
Query top-five co-occurrence lists, ranked by PMI, filtering candidates
to unigrams (filtering out bigrams), and filtering out co-occurrence
candidates with count less than five. Run queries from the YAML
specification in top-y.yaml
, using counts from
snli_stats/counts.pkl
, and write output to snli_stats/pmi.txt
:
python snli_query.py \
-k 5 \
--filter-to-unigrams \
--top-y-score-func pmi \
--min-count 5 \
snli_stats/counts.pkl top-y top-y.yaml snli_stats/pmi.txt
Run python snli_query.py --help
for more options.
Alternatively, run queries for all parameter configurations, in a loop:
bash snli_query_loop.bash
To change the default input paths and output directory, or change the
Python interpreter used to run snli_query.py
or other settings,
create a file named snli_query_loop_include.bash
with the following
contents and modify them as desired (and then run
snli_query_loop.bash
):
min_count=5
output_dir=snli_stats
python=python
extra_args='-k 5 --filter-to-unigrams --top-y-score-func pmi'
query_type=top-y
query_path=top-y.yaml
output_ext=.txt
input_dir=snli_stats
input_paths=`find "$input_dir" -type f -name '*.pkl'`
In the definition of the likelihood ratio Λ(C') in the paper (last equation on the second page, or page 75 in the proceedings), the summations should be products. The code and results use the correct definition.