-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
globally normalized models #960
Comments
Sure, I don't see why we shouldn't include this. |
This was referenced Nov 29, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Globally normalized models score sequences using the sum of unnormalized logits, as opposed to locally normalized ones which take a log_softmax at each token position before summing to compute the sequence score. They are strictly more expressive then locally normalized models (i.e. they can express a superset of probability distributions over sequences). It would take a one-line change in lm-evaluation-harness/lm_eval/base.py to allow evaluation of globally normalized models:
I just wanted to start a discussion to see if there is any interest and if there is a way we could make this an option for the evaluator. I can submit a pull request. (And I have some globally normalized models I'd like to share on hf-leaderboard ;).
The text was updated successfully, but these errors were encountered: