• Python 2.7
• PyTorch 0.1.12.post2 (GPU)

pip install -r requirements.txt

./install_kenlm.sh

Download the zip file from Google Drive, and copy it to the folder of code.

unzip acl18confidence_data_model.zip

./train_lm.sh ifttt
./train_lm.sh django

(This step can be skipped. The precomputed data have been cached in the zip file.)

# arg2: gpu id
./compute_metric.sh ifttt 0
./compute_metric.sh django 0

The intermediate results are saved to "data_model/[ifttt|django]/*.eval".

• Train a confidence scoring model.
• Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).

./train_scoring.sh ifttt
./train_scoring.sh django

Evaluate confidence backpropagation and attention-based method against inferred ground truth.

./eval_conf_bp_att.sh ifttt
./eval_conf_bp_att.sh django

• The implementation is based on OpenNMT-py.
• XGBoost: gradient tree boosting model.
• KenLM: language model.