Thank you very much for the contribution!

Would you be willing to test the scores of Llama 2 7b and Llama 1 7b with "Q:" and "A:" prompt against their reported scores in the paper? Unless you're aware of other notable papers that report they use "Question:" and "Answer:" as their prompt format, I think since we know this split is used by LLama it makes sense to match their evaluation setup.

Good point, it seems from what I can gather that PaLM and GPT-4 also use "Q:" and "A:" in their prompts so I agree it would make most sense if we use them as well. I have updated the code.

I have also fixed the normalization, articles at the start of the string weren't being removed properly. The normalization is the same as used by e.g. FiD for NQ and in the SQuAD evaluation code.

Here are the updated evaluation results with adjusted "Q:" and "A:" prompt and fixed normalization:

Hmm, when I run

python main.py --model hf-causal-experimental --batch_size auto --task nq_opee --model_args pretrained=huggyllama/llama-7b

I get only 5%:

hf-causal-experimental (pretrained=huggyllama/llama-7b), limit: None, provide_description: False, num_fewshot: 0, batch_size: auto
| Task  |Version|Metric|Value |   |Stderr|
|-------|------:|------|-----:|---|-----:|
|nq_open|      0|em    |0.0507|±  |0.0037|

Do you know why this might not match with the score you're reporting? I'm on the most recent commit of your PR branch.

EDIT: I do get 19.47% on 1-shot though:

|-------|------:|------|-----:|---|-----:|
|nq_open|      0|em    |0.1947|±  |0.0066|

These are with batch size = 8.

For the purpose of direct comparison with the results from the Llama paper I tried to reproduce their approach and prompt as closely as possible and used their task description in the prompt, i.e. "Answer these questions:" (see the Appendix A in the paper https://arxiv.org/pdf/2302.13971.pdf). Whether we use this kind of task description or not makes a much bigger difference in the 0-shot case, without it I saw scores which were similar to what you got.

Looking at your command it looks like you may have run the evaluation without the task description, so I think that would explain the large difference. Could you re-run the eval with the task description and see if scores match my numbers?

I simply used --description_dict descriptions.json and my descriptions.json just looks like this:

{
        "nq_open": "Answer these questions:"
}

Thanks, you're correct, I missed that you are passing a description_dict! Confirmed my results now match yours. In that case, I think this is good to merge--thanks very much for the contribution, it's great to finally have NQ in the repo!