Add Unit Tests / Style Tests

[sdtblck]

Describe the solution you'd like
Add unit tests to check:

• Code runs without error
• model outputs are consistent
• Code style (pep8) is enforced (https://gist.github.com/benjiyamin/e87a77c5d4d29e76b20c4176d96499f5)
• Code is documented

We should add these into github actions where possible (i.e if the unit tests can run on CPU rather than GPU)

[itsabhianant]

Hi @sdtblck,

I know how to create unit tests but I'm not very proficient in doing so. I have a basic idea of how unit test work and how to create one. I can help you resolve this issue but I will need support from you because I find this issue a great opportunity for getting more familiar with creating test cases and testing programs. If you are willing to help me solve this issue please do assign this issue to me and also explain this issue in brief.

I will be pleased if I get the opportunity to solve this issue as it will help me a lot.

Best,
Abhishek

[sdtblck]

Hi @itsabhianant ! Thanks for reaching out!
I can't really hold your hand through the process, but I can point you to some good resources.
I would recommend you start by looking at deepspeed's unit tests. They do quite a similar thing to what we'd like to do, and the codebase is also similar. https://github.com/microsoft/DeepSpeed/tree/master/tests

Specifically we may want to add some tests similar to the ones here: https://github.com/microsoft/DeepSpeed/tree/master/tests/model/Megatron_GPT2