-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fine tuning opus nmt ar-en using my own dataset #77
Comments
I do fine-tuning directly with MarianNMT. Maybe you could ask at the transformers git repository how to do finetuning with their library? If you use OPUS-MT models and marian-nmt then you would need the subword tokenisation on the fine-tuning data as well. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I want to fine tune the opus nt ar-en model using my own dataset, but I'm not sure what type of files my training data should be in? In the huggingface Marian tutorial (https://huggingface.co/docs/transformers/model_doc/marian) they just pass in lists of sentences, but I also read somewhere that I'm supposed to preprocess the data with Sentencepiece first. Or is sentencepiece "built in" into the arian tokenizer? All help is much appreciated.
The text was updated successfully, but these errors were encountered: