Implement arithmetic evaluations

[StellaAthena]

From the GPT-3 paper:

To test GPT-3’s ability to perform simple arithmetic operations without task-specific training, we developed a small
battery of 10 tests that involve asking GPT-3 a simple arithmetic problem in natural language:
• 2 digit addition (2D+) – The model is asked to add two integers sampled uniformly from [0, 100), phrased in
the form of a question, e.g. “Q: What is 48 plus 76? A: 124.”
• 2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from [0, 100); the
answer may be negative. Example: “Q: What is 34 minus 53? A: -19”.
• 3 digit addition (3D+) – Same as 2 digit addition, except numbers are uniformly sampled from [0, 1000)
• 4 digit addition (4D+) – Same as 3 digit addition, except uniformly sampled from [0, 10000).
• 4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from [0, 10000).
• 5 digit addition (5D+) – Same as 3 digit addition, except uniformly sampled from [0, 100000).
• 5 digit subtraction (5D-) – Same as 3 digit subtraction, except uniformly sampled from [0, 100000).
• 2 digit multiplication (2Dx) – The model is asked to multiply two integers sampled uniformly from [0, 100),
e.g. “Q: What is 24 times 42? A: 1008”.
• One-digit composite (1DC) – The model is asked to perform a composite operation on three 1 digit numbers,
with parentheses around the last two. For example, “Q: What is 6+(4*8)? A: 38”. The three 1 digit numbers
are selected uniformly on [0, 10) and the operations are selected uniformly from {+,-,*}.

In all 10 tasks the model must generate the correct answer exactly. For each task we generate a dataset of 2,000 random
instances of the task and evaluate all models on those instances. To spot-check whether the model is simply memorizing specific arithmetic problems, we took the 3-digit arithmetic problems in our test set and searched for them in our training data in both the forms " + =" and " plus ". Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000 subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers could have been memorized. In addition, inspection of incorrect answers reveals that the model often makes mistakes such as not carrying a “1”, suggesting it is actually attempting to perform the relevant computation rather than
memorizing a table

• Data processing code implemented
• Evaluation implemented

This should be modeled after the BoolQ task in lm_eval/tasks/suerglue.py

[VitamintK]

I'll take this!

[StellaAthena]

@VitamintK Hey Kevin, how's this coming? Still planning on doing it?

[VitamintK]

I am, but I've been putting it off and haven't started yet.

[StellaAthena]

No worries! Just wanted to check in and make sure it was on your radar. Do you think you'll be able to get it done in the next week or two? Finishing implementing the eval datasets is currently the major project blocker; we need to finish this before we can dedupe and we need to do that before we can start training models.

[VitamintK]

Yup, I'll try to get it done soon!

[StellaAthena]

@VitamintK What's happened? Is this something you have done? Should I assign it to someone else?

[VitamintK]

@StellaAthena hey, sorry for not following up with this until now, I am the worst. Finally got around to it, though!