Will there going to be checkpoints for masked language models?

[luffycodes]

Is there going to be support for MLMs (not just CLMs)?

[StellaAthena]

Possibly in the future, especially if there’s substantial interest in it from the academic community. We do plan on growing the suite over time with various configurations, the question is what’s the most impactful.

Depending on what you want to do, the RoBERTa models from this paper (which we sponsored the training of) could suit your needs.