To use DistributedSampler or not?

[abhishekkrthakur]

In pytorch-xla documentation: http:https://pytorch.org/xla/, it doesnt mention use of distributed sampler.
However, in the example : https://github.com/pytorch/xla/blob/master/test/test_train_mp_mnist.py , it says we should be using distributed samples.

xm.RateTracker() isnt mentioned in the documentation either.

Are both correct?

Also, Is there a way to use iterable datasets with distributed samplers?

[jysohn23]

You want to use distributed samplers when using the multiprocessing API (or TPU Pods training) since they don't share memory. So yes that example is correct.

Also yes, xm.RateTracker() is used in our examples.

Not sure if that'd work since to shard the dataset, which is what distributed sampler is doing, it needs to know the length of the entire dataset in advance. Check this discussion out: pytorch/pytorch#28743.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.