Replies: 2 comments 1 reply
-
Hello @stefan-it, the documentation uses |
Beta Was this translation helpful? Give feedback.
-
I would like to add to the conversation, and have some guidance and insights on what to look for when using adapters. I've been experimenting simple NER with distilbert models and it works great given our training and validation corpus, but as soon as I try with adapters, the performance and accuracy massively falls. |
Beta Was this translation helpful? Give feedback.
-
Hi everybody 🤗
I would like to perform experiments with first LM Adapters! So the main intention is, to use an existing multilingual LM (more precisely on of my hmBERT models) and train a LM Adapter. I would like to adapt new languages or new pretrain corpora for existing languages with e.g. less OCR errors or different domain (books instead of newspapers)...
My question is now about hyper-parameters that are recommended for pretraining on a) small corpora (in range of 1-2GB of text) and b) larger corpora (approx. 30GB of text).
So the example documentation uses
seq_bn
as adapter config. Is this config recommended for my use case 🤔Many thanks!
Beta Was this translation helpful? Give feedback.
All reactions