Skip to content

Issues: huggingface/tokenizers

ByteLevelBPETokenizer output seems weird
#203 opened Mar 24, 2020 by seyyaw
Open 2
Training a model from in-memory data
#198 by loicbarrault was closed Nov 28, 2020
Closed 1
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

32 Open 938 Closed
32 Open 938 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

BPE trainer ignoring special tokens.
#1616 opened Aug 16, 2024 by henrycharlesworth
2
.NET bindings
#1615 opened Aug 16, 2024 by sappho192
Can I use SentencePieceBPETokenizer to replace google/sentencepiece?
#1614 opened Aug 16, 2024 by npuichigo
4
Space after unnormalized token is added when use_fast=True for Llama tokenizer
#1613 opened Aug 14, 2024 by Butanium
6
RefMutContainer is unsound
#1612 opened Aug 13, 2024 by CheaterCodes
3
[test-infra] Enable Codecov for tokenizers
#1611 opened Aug 12, 2024 by hvaara
Token ID Out of Range & Indexing Assertion Errors During Training
#1609 opened Aug 9, 2024 by haseebrj17
4
Support for Golang now or support a cli for other languages?
#1601 opened Aug 7, 2024 by xuxiaoxia96
1
ValueError: The following model_kwargs are not used by the model: ['num_beans'] (note: typos in the generate arguments will also show up in this list)[06/Aug/2024 14:34:35]
#1596 opened Aug 6, 2024 by navdeep8990
1
[building on windows] onig_sys/oniguruma two or more data types in declaration specifiers
#1581 opened Jul 29, 2024 by louis030195
2
Risk of global variable memory leaks when calling train_from_iterator
#1579 opened Jul 24, 2024 by Yikai-Liao
Issue with SentencePieceUnigramTokenizer Handling Unknown Tokens
#1576 opened Jul 22, 2024 by Munikumar09
1
apply_chat_template api usage consult
#1575 opened Jul 22, 2024 by FanZhang91
Truncation performs slowly. Tokenizer firstly encodes long sequence and then truncates it. Feature Request
#1573 opened Jul 19, 2024 by galtimur
2
Tokenizer.from_bytes() not available in python bindings Feature Request Stale
#1567 opened Jul 11, 2024 by RamvigneshPasupathy
4
Custom fast PreTokenizer, ported via PyO3 to Python
#1566 opened Jul 10, 2024 by vandrw
2
Decode regression decoding performance
#1564 opened Jul 10, 2024 by daulet
10
Unable to Load Custom GPT2 Tokenizer - " data did not match any variant of untagged enum ModelWrapper at line 1 column 3193814" Error
#1562 opened Jul 5, 2024 by maghwa
3
Progress bar doesn't show in log file.
#1559 opened Jun 20, 2024 by amssljc
4
Encoding object stub doesn't include __len__
#1556 opened Jun 18, 2024 by thearchitector
4
Llama-3 offset-mapping needs fixing
#1553 opened Jun 14, 2024 by davidb-cerebras
6
[Bug?] Modifying normalizer for pretrained tokenizers don't consistently work Stale
#1552 opened Jun 12, 2024 by alvations
3
"Solution" to memory hogging in train_new_from_iterator with a hack
#1546 opened Jun 4, 2024 by morphpiece
7
Memory leak for large strings
#1539 opened May 23, 2024 by noamgai21
11
How to allow the merging of consecutive newline tokens \n when training a byte-level bpe tokenizer?
#1534 opened May 18, 2024 by liuslnlp
4
ProTip! Add no:assignee to see everything that’s not assigned.