explicitly set multiprocess start method to fork for cross-OS consist…

…ency

The method for launching a process can be "spawn", "fork", and "forkserver".
The default on Unix is fork, and the resulting process inherits all resources
from the parent process. Conversely, the default on Mac OS X/Windows is spawn,
which results in a minimal number of resources inherited by the child process.

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

scripts/finish_dedup_wiki40b.py

@@ -110,7 +110,7 @@ def _generate_examples(self, split):
  data_dir=args.data_dir)
 
 
- p = mp.Pool(96)
+ p = mp.get_context("fork").Pool(mp.cpu_count())
  i = -1
  for batch in ds:
  i += 1

scripts/load_dataset.py

@@ -74,7 +74,7 @@ def tok(x):
 
 fout = open(os.path.join(save_dir, dataset_name+"."+split), "wb")
 
-with mp.Pool(mp.cpu_count()) as p:
+with mp.get_context("fork").Pool(mp.cpu_count()) as p:
  i = 0
  sizes = [0]
  for b in ds: