[CLS] [SEP] bug

[dzabraev]

mmt/base/base_dataset.py

Line 343 in ef81f96

if not hasattr(self.tokenizer, "cls_token_ids"):

The class transformers.tokenization_bert.BertTokenizer does not have cls_token_ids. Here must be cls_token_id
Due to this mistake all text examples do not have SEP and CLS tokens.

[gabeur]

Good catch, thanks!
Fixing this bug seems to improve the results a bit. I will re-run pretraining and finetuning experiments and update the repo once the checkpoints are ready.

[gabeur]

I have pushed a fix, re-run the experiments and updated the checkpoints accordingly.