-
Notifications
You must be signed in to change notification settings - Fork 272
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why the speed is very slow? #180
Comments
i have encountered the same problem bro. Have you solved this issue? I nearly spent 2 hours for training one epoch on Voxceleb2 training set. And my gpu usage was utilized for one second while training single batch. I tried to debug to find out which it wastes too much time on reading/loading batch data. dont know if reading wav file stuck I/O channel or something bring the slow speed. |
@mcflyhu Hi have now much improved the speed as follows, Processing 14700 of 1076800:Loss 13.115565 TEER/TAcc 0.000% - 228.81 Hz I made modifications to enable distributed training, and I noticed the previous port was used by some other process also. So I have changed the --port and --distributed', dest='distributed', action='store_false' Note - I am using NVIDIA A10 GPU and T4 GPUS. |
@dimuthuanuraj Thanks for your immediate reply, |
I have experiencing several time speed is not going more than, 100Hz, I am using NvidiaA10 GPU with server. Could anyone please help me to figure this out?
Processing 1900 of 1076800:Loss 13.631720 TEER/TAcc 0.000% - 87.06 Hz
The text was updated successfully, but these errors were encountered: