Implement the StoryCloze evaluation

[StellaAthena]

From the GPT-3 paper

We next evaluate GPT-3 on the StoryCloze 2016 dataset [MCH+16], which involves selecting the correct ending sentence for five-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (with K = 70). This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [LDL19] but improves over previous zero-shot results by roughly 10%.

• Data processing code implemented
• Evaluation implemented

The evaluation code should be modeled after the interface in lm_eval/base.py and the example of the BoolQ task in lm_eval/tasks/suerglue.py

[cfoster0]

Form to get access is here:

https://cs.rochester.edu/nlp/rocstories/

[StellaAthena]

I have the download links and can provide them to anyone who DM’s me on Discord.

[jon-tow]

Implemented in #300