-
Notifications
You must be signed in to change notification settings - Fork 758
Issues: huggingface/tokenizers
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Can I use SentencePieceBPETokenizer to replace google/sentencepiece?
#1614
opened Aug 16, 2024 by
npuichigo
Space after unnormalized token is added when
use_fast=True
for Llama tokenizer
#1613
opened Aug 14, 2024 by
Butanium
Token ID Out of Range & Indexing Assertion Errors During Training
#1609
opened Aug 9, 2024 by
haseebrj17
Support for Golang now or support a cli for other languages?
#1601
opened Aug 7, 2024 by
xuxiaoxia96
[building on windows] onig_sys/oniguruma two or more data types in declaration specifiers
#1581
opened Jul 29, 2024 by
louis030195
Risk of global variable memory leaks when calling train_from_iterator
#1579
opened Jul 24, 2024 by
Yikai-Liao
Issue with
SentencePieceUnigramTokenizer
Handling Unknown Tokens
#1576
opened Jul 22, 2024 by
Munikumar09
Truncation performs slowly. Tokenizer firstly encodes long sequence and then truncates it.
Feature Request
#1573
opened Jul 19, 2024 by
galtimur
Tokenizer.from_bytes() not available in python bindings
Feature Request
Stale
#1567
opened Jul 11, 2024 by
RamvigneshPasupathy
[Bug?] Modifying normalizer for pretrained tokenizers don't consistently work
Stale
#1552
opened Jun 12, 2024 by
alvations
"Solution" to memory hogging in train_new_from_iterator with a hack
#1546
opened Jun 4, 2024 by
morphpiece
How to allow the merging of consecutive newline tokens \n when training a byte-level bpe tokenizer?
#1534
opened May 18, 2024 by
liuslnlp
Previous Next
ProTip!
Add no:assignee to see everything that’s not assigned.