Inverse Scaling Tasks?

[RylanSchaeffer]

Apologies if this has been asked before, but I couldn't find the answer in lm_evals/tasks or any issues. Are there plans to add Inverse Scaling (https://github.com/inverse-scaling/prize) into the lm-evaluation-harness?

[haileyschoelkopf]

Hasn't been asked before!

Supporting these tasks as originally implemented would be very nice! We ourselves probably won't have the bandwidth for it soon, but if anyone wishes to contribute them we'd be happy to assist and review.

[h-albert-lee]

That implementation looks interesting, do you mind if I try it?

[haileyschoelkopf]

Yes, that'd be fantastic if you're interested!

[h-albert-lee]

Thank you for assigning. I'll get to work soon!

[h-albert-lee]

To address any possible issues, I'm currently asking the inverse scaling slack if it's okay to implement these tasks. I will start implementing them as soon as they are approved.

[h-albert-lee]

@RylanSchaeffer @haileyschoelkopf The initial implementation is done, all that's left is to test that it produces results like the paper. I'll make a pull request once I've verified the results.

[RylanSchaeffer]

Awesome :) Cheers, Rylan Schaeffer

…

On Fri, Mar 8, 2024 at 6:25 PM Hanwool Albert Lee ***@***.***> wrote: @RylanSchaeffer <https://github.com/RylanSchaeffer> @haileyschoelkopf <https://github.com/haileyschoelkopf> The initial implementation is done, all that's left is to test that it produces results like the paper. I'll make a pull request once I've verified the results. — Reply to this email directly, view it on GitHub <#1442 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEHLCZBRHSEGU3IKRGC363YXJXLLAVCNFSM6AAAAABDOMNE6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWGY4TONZRGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[h-albert-lee]

@haileyschoelkopf
Hi! I compared the scores shown in the paper with the scores obtained through lm-eval-harness using OPT 125M through 6.7B models, and there is a slight difference. I was wondering if lm-eval-harness tolerates a certain amount of score difference?

[h-albert-lee]

Or maybe I'll use the code utilized in the evaluation(of inverse scaling prize) as a custom metric. I didn't get anything from the inverse-scaling team, but I did find some related work in the authors' github.

[haileyschoelkopf]

@h-albert-lee That’s great progress!

Would you be able to open a PR with your implementation and resulting scores so we can discuss there? It’s hard to say without being able to look at the implementation differences/concrete numbers.

[h-albert-lee]

@haileyschoelkopf Thanks a lot!, I'll apply the pre-commit and post a pull request with my experimental results soon.