Skip to content

Adaptive floating-point based numerical format for resilient deep learning

Notifications You must be signed in to change notification settings

ttambe/AdaptivFloat

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 

Repository files navigation

AdaptivFloat: A Floating-Point Based Data Type for Resilient Deep Learning Inference

AdaptivFloat is a floating-point inspired number representation format for deep learning that dynamically maximizes and optimally clips its available dynamic range, at a layer granularity, in order to create faithful encoding of neural network parameters.

AdaptivFloat consistently produces higher inference accuracies compared to block floating-point, uniform, IEEE-like float or posit encodings at low precision (≤ 8-bit) across a diverse set of state-of-the-art neural network topologies.

The table below shows the impact of weight bit compression on the BLEU score (WMT'17 En-to-De) of the Transformer model post-training quantization / post-quantization aware retraining. More results on the paper (see reference below):

# Bits Float BFP Uniform Posit AdaptivFloat
16 27.4 / 27.4 27.4 / 27.4 27.4 / 27.4 27.4 / 27.5 27.4 / 27.6
8 27.2 / 27.5 26.3 / 27.3 27.3 / 27.4 27.3 / 27.5 27.3 / 27.7
7 27.1 / 27.5 16.9 / 26.8 26.0 / 27.2 27.3 / 27.4 27.3 / 27.7
6 26.5 / 27.1 0.16 / 8.4 0.9 / 23.5 26.7 / 27.2 27.2 / 27.6
5 24.2 / 25.6 0.0 / 0.0 0.0 / 0.0 25.8 / 26.6 26.4 / 27.3
4 0.0 / 0.0 0.0 / 0.0 0.0 / 0.0 0.0 / 0.0 16.3 / 25.5

Algorithm

The base algorithm can be found in the script adaptivfloat.py. It can be easily invoked in any ML framework (PyTorch, TensorFlow, etc.) with a Python backend. The example below shows how one can quantize DNN layers' weights to AdaptivFloat format in PyTorch. The user just needs to be specify the desired quantization bit width f_bits of the tensor and its number of floating-point exponent bits f_exp.

import torch
import numpy
from adaptivfloat import quantize_adaptivfloat

for parameter in self.model.named_parameters():
    params_np = parameter.cpu().data.numpy()
    params_adaptivfloat = quantize_adaptivfloat(params_np, self.f_bits, self.f_exp, bias = None)
    parameter.data = torch.from_numpy(params_adaptivfloat).float().cuda()

Citation

If you find this resource useful, please consider citing the following paper:

@article{Tambe2019AdaptivFloatAF,
  title={AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference},
  author={Thierry Tambe and En-Yu Yang and Zishen Wan and Y. Deng and V. Reddi and Alexander M. Rush and D. Brooks and Gu-Yeon Wei},
  journal={ArXiv},
  year={2019},
  volume={abs/1909.13271}
}

About

Adaptive floating-point based numerical format for resilient deep learning

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages