Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torchtext.data.Iterator loading data too slow #668

Closed
zml24 opened this issue Dec 19, 2019 · 3 comments
Closed

torchtext.data.Iterator loading data too slow #668

zml24 opened this issue Dec 19, 2019 · 3 comments
Labels
legacy

Comments

@zml24
Copy link

zml24 commented Dec 19, 2019

❓ Questions and Help

Description
Hey Dude, I am running a machine translation task with torchtext.datasets.IWSLT. However, I found that the CPU usage is always high and GPU usage is always low. When using torch.utils.data.dataloader, I can use num_workers and pin_memory to deal with this problem. But I can't deal with this problem with torchtext.data.Iterator or torchtext.data.BucketItertor.

How can I accelerate my data loading speed via torchtext?
@zhangguanheng66
Copy link
Contributor

zhangguanheng66 commented Dec 19, 2019 •
edited
Loading

Yes, we will re-write the translation datasets later so they will be compatible with torch.utils.data.dataloader (see discussions #664)

@zhangguanheng66 zhangguanheng66 mentioned this issue Dec 19, 2019
Open
@lizitong67
Copy link

How to solve this problem？？

@Nayef211
Copy link
Contributor

Closing this since torchtext.data.Iterator has been deleted and is no longer available in the latest releases of our library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
legacy
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@zhangguanheng66 @zml24 @Nayef211 @lizitong67