Implement the WSC273 Winograd Schemas Challenge evaluation

[StellaAthena]

From the GPT-3 paper

The Winograd Schemas Challenge [LDM12] is a classical task in NLP that involves determining which word a pronoun refers to, when the pronoun is grammatically ambiguous but semantically unambiguous to a human. Recently fine-tuned language models have achieved near-human performance on the original Winograd dataset, but more difficult versions such as the adversarially-mined Winogrande dataset [SBBC19] still significantly lag human performance. We test GPT-3’s performance on both Winograd and Winogrande, as usual in the zero-, one-, and few-shot setting.

On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method described in [RWC+19]. Note that this setting differs slightly from the WSC task in the SuperGLUE benchmark, which is presented as binary classification and requires entity extraction to convert to the form described in this section. On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human performance. We note that contamination analysis found some Winograd schemas in the training data but this appears to have only a small effect on results (see Section 4).

On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a fine-tuned RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a fine-tuned high capacity model (T5), and human performance on the task as reported by [SBBC19] is 94.0%.

• Data processing code implemented
• Evaluation implemented

The evaluation code should be modeled after the interface in lm_eval/base.py and the example of the BoolQ task in lm_eval/tasks/suerglue.py

[cfoster0]

I'm having trouble finding an official source for this, but it appears that this may be a BERT version of the WSC273 dataset:

https://github.com/vid-koci/bert-commonsense/blob/master/data/wsc273.txt

[cfoster0]

The original was hosted on a Google Cloud Storage bucket. gs:https://commonsense-reasoning/reproduce/commonsense_test/wsc273.json Unfortunately, that is no longer available.

It likely is extant at the link below. The examples in Trinh and Le's paper are included.
https://git.cse.msu.edu/bakerb15/nlp-final-project/raw/master/Winogard/reproduce/commonsense_test/wsc273.json

[cfoster0]

Reopening. In case anyone else ends up confused (as we were) there are 4 different Winograd schema datasets that will be in this harness:

1. WSC273, which is a set of 273 Winograd schemas. That's what this issue is about.
2. Winogrande (XL), which is a set of 44k Winograd schemas obtained through an adversarial mining scheme. Add Winogrande dataset #45
3. WNLI from the GLUE benchmark, which recasts the whole thing as an NLI task. LM Eval Refactor; GPT-3; GLUE tasks #3
4. WSC from the SuperGLUE benchmark, which pulls from both the original set plus some from an organization called Commonsense Reasoning. SuperGLUE part 1 #4