Implement the Penn Tree Bank evaluation

[StellaAthena]

We calculate zero-shot perplexity on the Penn Tree Bank (PTB) [MKM+94] dataset measured in [RWC+19]. We omit the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the one-billion word benchmark due to a high fraction of the dataset being contained in our training set. PTB escapes these issues due to predating the modern internet. Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplaexity of 20.50. Note that since PTB is a traditional language modeling dataset it does not have a clear separation of examples to define one-shot or few-shot evaluation around, so we measure only zero-shot.

[sdtblck]

not free - https://catalog.ldc.upenn.edu/LDC99T42 / https://stackoverflow.com/questions/8949517/is-there-any-treebank-for-free
I suggest leaving this one out

[StellaAthena]

Dropping because it isn’t free.

[leogao2]

We received confirmation from OA that they use the version of PTB with all the <unk>s. So we are going to be using PTB now.

[StellaAthena]

We received confirmation from OA that they use the version of PTB with all the s. So we are going to be using PTB now.

What does this mean?

[leogao2]

Edited comment for clarity, apparently github dropped the <unk> because it thought it was a html tag or something, and i never noticed.

[leogao2]

eric hallahan is working on this