-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
CombinedLoader does not work in DDP when using max_size_cyle
option
#10373
Labels
Comments
Dear @ant0nsc, Thanks for raising this issue, it was in fact never supported. Currently looking into it. Best, |
Thanks @tchaton ! is there a way how we can work around the issue in the meantime? |
12 tasks
Wow, that was swift, thanks @tchaton ! I tried on my small repro, it works just fine now. |
This was referenced Jan 26, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
馃悰 Bug
When using a
CombinedLoader
with themax_size_cycle
option and DDP, all the GPUs get all validation data.This bug is related to #7013 - however, the fix in PR #7102 only affect the default
min_size
option of theCombinedLoader
@tchaton ?
To Reproduce
Repro
Expected behavior
For the above repro, the validation data has length 8. I would expect that each of the 2 GPUs only get 4 batches each, but in fact they get 8 batches.
Environment
- GPU:
- Tesla K80
- Tesla K80
- available: True
- version: 10.2
- numpy: 1.21.2
- pyTorch_debug: False
- pyTorch_version: 1.8.0
- pytorch-lightning: 1.5.0
- tqdm: 4.62.3
- OS: Linux
- architecture:
- 64bit
-
- processor: x86_64
- python: 3.7.3
- version: 18.04.1-Ubuntu SMP Wed Jul 28 23:14:18 UTC 2021
Additional context
The text was updated successfully, but these errors were encountered: