llm int8 integration into Megatron part of metaseq

[erichan1]

Add LLM int8 to Megatron.
Steps to use

1. Flip on QUANTIZED_INFERENCE in layers.py
2. Comment out _log_weight_stats here. Not sure if this is simply buggy or that the weight stats is unhappy if we don't initialize the weights. Potentially a divide by zero issue.
3. Run thru metaseq (or whatever runner of your choice)

Why use hooks for load_state_dict?

• tl;dr it's basically a way to stream weights linear by linear from CPU=>GPU.
• I had a problem where I started with an fp16 checkpoint, but I wanted to load it into less GPUs than it was trained on. FP16 wouldn't fit into GPU memory. Tried fiddling with offline conversion, but it was messy. So fundamentally, we want to create fp16 weights on CPU, then only convert to int8 as we send to GPU.
• pre_hook allows us to not blow up CPU memory by creating all weights at once (eg during initial model creation). We only create CPU-side weights when a particular linear has load_state_dict called on it. That CPU weight is immediately destroyed when we send to GPU.
• After we load in fp16 weight onto CPU during load_state_dict, the post hook takes care of converting to int8 and sending to GPU.

[erichan1]

Set QUANTIZATION_LEVEL here for different quantization types