compute perplexity from cross entropy

[winglian]

No description provided.

[casper-hansen]

If we want this to match other libraries, we probably want to do something like

ppl = exp(sum(loss_all_batches) / num_of_batch)

Huggingface also has its own. I think you can set batch_size=512

from evaluate import load
perplexity = load("perplexity", module_type="metric")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return perplexity.compute(predictions=predictions, model_id='llama')