-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to compute the perplexity only on the answer? #1370
Comments
Maybe doc_to_text: !function utils.doc_to_text # result is {{prompted_input}}
doc_to_target: "{{prompted_input}}\n{{solution}}" is correct? |
OK, It is not correct too, |
You should be able to do this without a custom the relevant bits:
|
Thank you for reply. I use the config below like Lambada: group:
- math_word_problems_ppl
task: minerva_math_algebra_ppl
dataset_path: EleutherAI/hendrycks_math
process_docs: !function utils.process_docs
dataset_name: algebra
output_type: loglikelihood
training_split: train
test_split: test
doc_to_text: !function utils.doc_to_text
doc_to_target: "\n{{solution}}"
should_decontaminate: true
doc_to_decontamination_query: "{{prolem}}"
metric_list:
- metric: perplexity
aggregation: perplexity
metadata:
version: 1.0 And my
It almost get overflow... When I change the aggregation to
|
Ah, I think that this is not normalizing perplexity by length, possibly! I believe the following code snippet may fix this: https://github.com/EleutherAI/lm-evaluation-harness/blob/1554066c9532a7a1cf171b02a5ddaa5bb38f1b78/lm_eval/api/task.py#L1041C1-L1047C14
by
if you use
in the config. If this is indeed the solution, I will push a fix for this, for others who may want to calculate normalized target loglikelihood. |
Thank you!! Yes, I think it is not normalized too, but I am not familiar with the code.
|
Yes, I think this looks quite reasonable! Very similar to our results on Llemma for this eval. |
I am trying to calculate the perplexity on
minerva_math
, here are my task yaml config.The
doc_to_text
and theprocess_docs
functions are the same as the minerva_math.The
process_results
function is below like wikitext:My script is:
When I get deep into the
vllm_causallms.py
, the_loglikelihood_tokens
function'sinp
only has thetokenized_target
, thecontext_enc
is[2]
. If I have nothing wrong, I think the model didn't see thedoc_to_text
but only see thedoc_to_target
?However, I want to compute the perplexity like LLM's SFT procedure:
We only compute the model's output on labels. Just like this tutorial: https://huggingface.co/docs/transformers/main/en/perplexity
The text was updated successfully, but these errors were encountered: