Implement the DROP evaluation

[StellaAthena]

From the GPT-3 paper

Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19]. On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [LXL+17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

• Data processing code implemented
• Evaluation implemented

The evaluation code should be modeled after the interface in lm_eval/base.py and the example of the BoolQ task in lm_eval/tasks/suerglue.py

[cfoster0]

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/drop

[anishthite]

added in ec4d361 , ended up not using HuggingFace's implementation since they only included answer spans as labels, leaving out a ton of labels

[anishthite]

Also, should we prepend 'Passage: ' for each passage? It seems OA did not do this, not sure if it was intentional.

[StellaAthena]

My vote is “no” because I see no reason to do so.

[StellaAthena]

@anishthite I don't see a line for this dataset in lm_eval/tasks/init.py. Am I missing something, or is this not quite finished?

[StellaAthena]

@jon-tow How is the work on this coming?