Train M -> F pronoun interventions on selected Pythia models

[haileyschoelkopf]

(Re)train pythia models with the last 7% of training data adjusted to have all female pronouns.

• s3:https://s-eai-neox/pythia/1.3B_dedup/global_step66500 trained for last 5k steps with intervened data
• s3:https://s-eai-neox/pythia/19M_deduped/global_step133000 trained for last 10k steps with intervened data
• s3:https://s-eai-neox/pythia/350M_dedup/global_step66500 trained for last 5k steps with intervened data
• s3:https://s-eai-neox/pythia/6.7B_deduped_new/global_step133000 trained for last 10k steps with intervened data

All intervened models should be evaluated on the same benchmarks as #16 for all the saved checkpoints post-intervention. All saved intervened checkpoints should also be evaluated on the same benchmarks as chosen in #51 .

If get meaningful numbers from the above and have evaluated all the above models:

• s3:https://s-eai-neox/pythia/1.3B_dedup/global_step66500 trained for last 10k steps with intervened data

[ankit-bhattarai]

I have started working on this, I will start with 19M first and then move onto the bigger models!

[haileyschoelkopf]

Thanks for being willing to help on this @ankit-bhattarai ! We've now done these items, please let me know if you need any assistance and want to experiment with different interventions.