Different fewshot with and w/o accelerate DDP

[baberabb]

The fewshot splits are different for the same doc_id when using/not using DDP on accelerate.

[baberabb]

Edit:

The issue is that each DDP process runs its own instance of the rnd. Could have the rnd no longer be stateful and offset the seed by doc_id on each call. But if there's a way to share it across the processes, that would be better.

lm-evaluation-harness/lm_eval/api/samplers.py

Lines 66 to 71 in 97a67d2

	def sample(self, n):
	"""
	Draw `n` samples from our fewshot docs. This method should be overridden by subclasses.
	"""
	return self.rnd.sample(self.docs, n)