Minimal Implementation of Paper Deep Encoder Shallow Decoder with Pytorch nn.Transformer and Huggingface Tokenizer.

**Tested with 2.4 GHz 8-Core Intel Core i9

from transformers import PreTrainedTokenizerFast
from model import DeepShallowConfig, DeepShallowModel # fetch from model.py file

# source and target tokenizer
korean_tokenizer = PreTrainedTokenizerFast.from_pretrained("snoop2head/Deep-Shallow-Ko")
english_tokenizer = PreTrainedTokenizerFast.from_pretrained("snoop2head/Deep-Shallow-En")

# Korean(source) -> English(target) translation with pretrained model
config = DeepShallowConfig.from_pretrained("snoop2head/Deep-Shallow-Ko2En")
model = DeepShallowModel.from_pretrained("snoop2head/Deep-Shallow-Ko2En", config=config)

# English(source) -> Korean(target) translation with pretrained model
config = DeepShallowConfig.from_pretrained("snoop2head/Deep-Shallow-En2Ko")
model = DeepShallowModel.from_pretrained("snoop2head/Deep-Shallow-En2Ko", config=config)

Manipulate config.yaml for source language & target langauge setting or number of inference sample for the output.

python inference.py # creates translation output as csv file on result folder
python metrics.py # calculates bleu score based on csv output from inference.py

For customized training, change arguments in config.yaml.

python train.py # trains with arguments on config.yaml

Datasets were proviced by AIHub, which were Korean - English pairs.

Parallel corpus shape of source langauge and target language pairs.

Maximum Token Length was selected as 62 which is 95% percentile of the target dataset.

Source Language(Korean) Token Length Distribution	Target Language(English) Token Length Distribution

• Knowledge distllation is yet to be applied.
• Package seq2seq transformer model to huggingface hub.
• Make Machine Translation Model class similar to MarianMTModel in order to be trainable with huggingface trainer.
• Get BLEU score from the test set.
• Compare BLEU score from Google Translate, Papago, Pororo on the test set(or test set which is out of domain).
• Enable batch inferencing on GPU
• For tasks such as Named Entity Relation or Relation Extraction, enable entity marker to wrap around even after translation
• Inference with Beam Search

@article{Kasai2020DeepES,
  title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation},
  author={Jungo Kasai and Nikolaos Pappas and Hao Peng and J. Cross and Noah A. Smith},
  journal={ArXiv},
  year={2020},
  volume={abs/2006.10369}
}

inproceedings{opennmt,
  author    = {Guillaume Klein and
               Yoon Kim and
               Yuntian Deng and
               Jean Senellart and
               Alexander M. Rush},
  title     = {OpenNMT: Open-Source Toolkit for Neural Machine Translation},
  booktitle = {Proc. ACL},
  year      = {2017},
  url       = {https://doi.org/10.18653/v1/P17-4012},
  doi       = {10.18653/v1/P17-4012}
}

• LANGUAGE TRANSLATION WITH NN.TRANSFORMER AND TORCHTEXT
• The Annotated Transformer