accuracy precision

[lernerjenny]

Hi, running the evaluation I get acc with 1 decimal digit and I was looking for a way to see more precise results.

{'results': {'arc_easy': {'acc,none': 0.3, 'acc_stderr,none': 0.15275252316519464, 'acc_norm,none': 0.1, 'acc_norm_stderr,none': 0.09999999999999999, 'alias': 'arc_easy'}}

I am running this:

task_name = ['arc_easy']
limit=10
model_args = 'pretrained=EleutherAI/pythia-14m,dtype=float32,device=cpu'
bootstrap_iters=0

from lm_eval import tasks, evaluator, utils, api
tasks.initialize_tasks()

e1 = evaluator.simple_evaluate(
        model=model,
        tasks=task_name,
        limit=limit,
        model_args=model_args,
        bootstrap_iters=bootstrap_iters,
    )

Is there a way to show more digits after the decimal point?

[lernerjenny]

I found the problem - although I think it should be better documented:
limit=10 causes this issue.
but this setting appears in the example https://github.com/EleutherAI/lm-evaluation-harness/blob/main/examples/lm-eval-overview.ipynb and it is not pointed there what it means and what are the outcomes.
It only appears as a warning in the log: "--limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT."

[LSinev]

"PR welcome" ;)

[haileyschoelkopf]

Hi @lernerjenny , --limit 10 causes only the first 10 documents to be evaluated (and for this reason, is only recommended for debugging purposes). The reason only one digit is therefore reported is because it's only possible to get an accuracy with 0.1 increments when you have 10 docs--we don't right-pad with zeros in this case.