Skip to content

Issues: huggingface/datatrove

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

20 Open 49 Closed
20 Open 49 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

JSONL loading slow when using megawarcs
#206 opened Jun 3, 2024 by cryptowooser
Memory overflow issue with long-context data using datatrove
#204 opened May 31, 2024 by justHungryMan
1
The naming in gopher_quality_filter seems to be incorrect
#202 opened May 30, 2024 by ryan-minato
Minhash mersenne hashing overflow issues.
#198 opened May 28, 2024 by Apsod
2
How to deduplicate multiple datasets?
#196 opened May 28, 2024 by canghaiyunfan
2
Migrate word tokenizer download functions to process locked download
#190 opened May 22, 2024 by hynky1999
Fastwarc reader
#182 opened May 13, 2024 by jordane95
Dependency resolving issue installing from source
#173 opened May 4, 2024 by its5Q
4
Periodical logging of stats
#171 opened May 3, 2024 by rantav
2
OpensearchWriter
#166 opened May 2, 2024 by rantav
2
PineconeWriter
#165 opened May 2, 2024 by rantav
Memory overhead in multiprocessing
#161 opened Apr 24, 2024 by jordane95
2
Potential issues in substring dedup bug Something isn't working
#121 opened Mar 12, 2024 by jordane95
10
Support int32 in substring dedup
#118 opened Mar 8, 2024 by jordane95
4
Flexibility in minhash dedup by index
#110 opened Feb 28, 2024 by jordane95
8
In-file parallelism enhancement New feature or request
#74 opened Jan 31, 2024 by jordane95
Spark support enhancement New feature or request
#70 opened Jan 30, 2024 by jordane95
1
Implementation of line-wise corrections enhancement New feature or request
#69 opened Jan 30, 2024 by jordane95
1
Support Ray as executor enhancement New feature or request
#62 opened Jan 23, 2024 by c21
2
Supporting Apache Beam enhancement New feature or request
#58 opened Jan 22, 2024 by sayakpaul
ProTip! Follow long discussions with comments:>50.