This is the official repo for the Findings of EMNLP 2020 paper: The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation .

This repository contains three types of test suite:

• lexical ambiguity

• contextless syntactic ambiguity

• contextual syntactic ambiguity