social_iqa choices do not use actual answers

[ozgurcelik]

Looking at the logged samples

{
"doc_id": 0,
"doc": {
"context": "Tracy didn't go home that evening and resisted Riley's attacks.",
"question": "What does Tracy need to do before this?",
"answerA": "make a new plan",
"answerB": "Go home and see Riley",
"answerC": "Find somewhere to go",
"label": "3"
},
"target": 2,
"arguments": [
[
"Q: Tracy didn't go home that evening and resisted Riley's attacks. What does Tracy need to do before this?\nA:",
" {{answerA}}"
],
[
"Q: Tracy didn't go home that evening and resisted Riley's attacks. What does Tracy need to do before this?\nA:",
" {{answerB}}"
],
[
"Q: Tracy didn't go home that evening and resisted Riley's attacks. What does Tracy need to do before this?\nA:",
" {{answerC}}"
]
]

We can see that Siqa doesn't use the answers properly. How can this be fixed? This is the yaml file

task: social_iqa
dataset_path: social_i_qa
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "Q: {{context}} {{question}}\nA:"
target_delimiter: " "
doc_to_choice:

• "{{answerA}}"
• "{{answerB}}"
• "{{answerC}}"
  doc_to_target: "{{ (label|int) - 1 }}"
  metric_list:
• metric: acc
  aggregation: mean
  higher_is_better: true
  metadata:
  version: 0.0

@haileyschoelkopf This can explain Accuracy not matched for Llama1-7b #1294 for Siqa. Since the choices are like {{answerC}}, the model is equally likely to predict any one of the choices leading ~0.33 acc.

[haileyschoelkopf]

switching to

doc_to_choice: "{{[answerA, answerB, answerC]}}" I believe should resolve this.

[ozgurcelik]

Yes, it does. Thank you very much.