Try to reproduce experiments in Table 1 in paper "Up or Down? Adaptive Rounding for Post-Training Quantization" According to the descriptions in paper, we:

• use SQNR observer (the class utilities.observers.MySQNRObserver), we modified C codes in AIMET into python
• fuse batch normalization into convolution (as described in Section 5 Experimental setup)
• use symmetric 4-bit weight quantization (as described in Section 5 Experimental setup)
• use per-tensor as quantization scheme (NOT per-channel)
• quantized first layer as described in paper
• apply stochastic rounding described in the paper "Deep Learning with Limited Numerical Precision"

Results in paper:

Our results:

We tried several times and found that stochastic rounding is NOT better than nearest rounding. It seems weired since stochastic rounding has about 50% chance to be better than nearest rounding as shown in paper.

hydra-core             1.2.0
pytorch-lightning      1.8.4.post0
torch                  1.10.1+cu102
torchaudio             0.10.1+cu102
torchmetrics           0.11.0
torchvision            0.11.2+cu102

Run experiments python main.py

Several settings can be tried:

• symmetric/asymmetric scheme
• with/without fusing BN into Conv
• quantized first layer or all layers
• per-channel or per-tensor
• rounding scheme, e.g. round for nearest rounding; ceil; floor; stochastic for stochastic rounding
• int8 or int4

See doc/analyze_rounding.xlsx