Use train data for computing normalization

[FabienRoger]

No description provided.

[lauritowal]

@FabienRoger @norabelrose
Seems like deberta performs worse after these changes:

for boolq for it was 70% before, now:

model,prefix,method,prompt_level,train,test,accuracy,std,language_model_type,layer,loss
deberta-v2-xxlarge-mnli,normal,ccs,all,boolq,boolq,0.551499992609024,0.051188381401737586,encoder,-1.0,0.37969053238630296
deberta-v2-xxlarge-mnli,normal,lr,all,boolq,boolq,0.6075,0.06677387213573883,encoder,-1.0,0.0"

for imdb for it was 89% before, now:

model,prefix,method,prompt_level,train,test,accuracy,std,language_model_type,layer,loss
deberta-v2-xxlarge-mnli,normal,ccs,all,imdb,imdb,0.8411666552225748,0.15101562370137744,encoder,-1.0,0.8821464618047078
deberta-v2-xxlarge-mnli,normal,lr,all,imdb,imdb,0.9106666666666665,0.07123591478710409,encoder,-1.0,0.0

We should check what is happening there (?)

[FabienRoger]

for boolq for it was 70% before, now:

I'm not finding that. I got 59% for both.

for imdb for it was 89% before, now:

I'm not finding that. I got 0.88 for both runs.

[lauritowal]

@FabienRoger Thanks for running that too! Are you sure that you are not loading an older model? What model type are you using? I'll try it out again, later.

[FabienRoger]

deberta-v2-xxlarge-mnli,normal,ccs,all,boolq,boolq,0.5864999830722809,0.0585042759828347,encoder,-1.0,0.9317224085330963

[FabienRoger]

I fixed a dumb bug and now the results are
On boolq, drop from 65% to 62%
On imdb, stays at 88%

[norabelrose]

LGTM