daormar / thot Star 50 Code Issues Pull requests Thot toolkit for statistical machine translation python c-plus-plus machine-learning natural-language-processing statistics tokenizer machine-translation artificial-intelligence shell-script pattern-recognition statistical-machine-translation detokenizer recaser Updated Nov 11, 2022 C++
littinrajan / detokenize Star 1 Code Issues Pull requests De-Tokenize is a Python package which provides fast, accurate structuring of tokens back to original sentence form python nlp python3 detokenizer detokenize littinrajan Updated Mar 5, 2024 Python
yas-sim / openvino_tokenizers_sample_codes Star 0 Code Issues Pull requests Demonstrates how to convert a tokenizer into OpenVINO IR model and how to integrate it into an NLP model. python nlp natural-language-processing sentiment-analysis tokenizer ner detokenizer openvino large-language-models llm Updated Jul 2, 2024 Jupyter Notebook
daormar / thot_preproc_utils Star 0 Code Issues Pull requests Pre/post processing utilities for the Thot translation toolkit. tokenizer detokenizer recaser Updated Mar 7, 2018 Python