Create an evaluation that measures a model's ability to remember specifics about texts in it's dataset?

[mrconter1]

Hi!

Would it make sense to create an evaluation that measures the model's ability to recall specifics about the data it has been trained on? I am thinking about putting together an evaluation that basically tests this on distinct strings that almost certainly exist in its dataset. Take this string for instance (which can't be found by Google):

In some research, the dosage went as excessive as 600 mg oregano oil per

Which is found in https://data.commoncrawl.org/crawl-data/CC-MAIN-2019-47/segments/1573496664437.49/wet/CC-MAIN-20191111191704-20191111215704-00000.warc.wet.gz.

If I ask GPT-4 to do the following:

The following are exempt from a specific online web page. What would be the next word in this exempt? Only reply with the next word!

"In some research, the dosage went as excessive as 600 mg..."

it fails:

---

It feels like this could be useful because it would perhaps demonstrate that the model knows exactly what it has read. If it, for instance, got a high score on this evaluation you would be able to train it on a code base and it would be able to recite it word by word. It's like a metric of memorization. My questions to you are:

1. Do you see any value in this?
2. Would it perhaps make more sense to ask it where it read it (assuming the text just exists on one page)? You would basically provide it with a unique string and ask it to reply with the URL on which I can find this string.

Regards, Rasmus

[jwang47]

This could be an interesting eval, please feel free to open a PR to add it. It might be interesting to also ask it where it read the string, so it may hallucinate on that but might still be interesting to measure.