A way to evaluate GPT models

[qrdlgit]

One of the things that's interesting to me is how well GPT can autonomously improve AI code.

I created this example here https://github.com/qrdlgit/gpt and it gets to about 0.972 in classifier accuracy after about 5 runs.

The ML problem is fairly basic, but it's just used as an example here. More sophisticated problems can be used, though some attention to the max context needs to be made.

Also, using problems which are unique and not easily solved via training data would be required for fair evaluation. Ideally a broad cross section of different problems would be useful to capture a more accurate assessment.

One of the next things I'm going to work on is pushing this up into RLHF (RLCF?) by defining a reward model specific to this domain.. Ie, does the response compile and run, provide intelligible metrics, was there an improvement, etc.