Can I provide custom data and continue training Pythia on this new data?

[GeorgiAngelov]

Can I provide custom data and continue training Pythia on this new data? I have data in a specific domain + language and I would like to use Pythia as the base model and simply continue training it with this additional new data. Is this how this would work or do I need to completely start from scratch ( i.e. traning a Pythia from the ground up with this new data ) ?

[StellaAthena]

Yes you can! You’ll need to either use a pre-final checkpoint or warm up the learning rate again, as the learning rate for the final checkpoint is very small. However you can absolutely finetune Pythia models, and many people have already done so both within EleutherAI and at other orgs.