Increase test coverage

[sweinbach]

This pull request does the following:

• executing tests in a distributed environment
• add test configs to test more configuration options
• add requirement for text coverage report and a small test readme

[sweinbach]

Converting the pull request to draft for the following reasons:

• Test coverage is not optimal (though most likely not the best metric to assess test quality). I am happy to include other suggested configs.
  

• I am getting a failed testcase for loading the state dict for one bit adam. I would appreciate help in validating that the testcase works as intendend and (potentially) in the fix of the error.

checkpoint_name, state_dict = model.load_checkpoint(neox_args.load,
  File "/home/aleph/repos/neox/gpt-neox-copy/src/deepspeed/deepspeed/runtime/engine.py", line 1476, in load_checkpoint
    load_path, client_states = self._load_checkpoint(load_dir,
  File "/home/aleph/repos/neox/gpt-neox-copy/src/deepspeed/deepspeed/runtime/engine.py", line 1519, in _load_checkpoint
    self.optimizer.load_state_dict(checkpoint['optimizer'])
  File "/home/aleph/repos/neox/gpt-neox-copy/src/deepspeed/deepspeed/runtime/fp16/onebit/adam.py", line 285, in load_state_dict
    if self.state[self.param_groups[0]['params'][0]]['step'] < self.freeze_step:
KeyError: 'step'

[sweinbach]

I am quite happy with the current status. Configuration files are used for a number of testcases. This should enable adding new testcases easily.

When reviewing please pay special attention to the commits starting with "fix ...". These fix minor issues and affect the codebase outside of test cases.

tests/Readme.md gives a short overview on how to run tests and also view the coverage report.

All testcases pass at the moment:

Coverage is far from100%. But a reasonable amount of code is covered.