Searching for Low-Bit Weights in Quantized Neural Networks

09/18/2020
by   Zhaohui Yang, et al.
7

Quantized neural networks with low-bit weights and activations are attractive for developing AI accelerators. However, the quantization functions used in most conventional quantization methods are non-differentiable, which increases the optimization difficulty of quantized networks. Compared with full-precision parameters (i.e., 32-bit floating numbers), low-bit values are selected from a much smaller set. For example, there are only 16 possibilities in 4-bit space. Thus, we present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately. In particular, each weight is represented as a probability distribution over the discrete value set. The probabilities are optimized during training and the values with the highest probability are selected to establish the desired quantized network. Experimental results on benchmarks demonstrate that the proposed method is able to produce quantized neural networks with higher performance over the state-of-the-art methods on both image classification and super-resolution tasks.

READ FULL TEXT
research
01/31/2023

Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance

We introduce a quantization-aware training algorithm that guarantees avo...
research
06/26/2022

CTMQ: Cyclic Training of Convolutional Neural Networks with Multiple Quantization Steps

This paper proposes a training method having multiple cyclic training fo...
research
12/06/2019

Sampling-Free Learning of Bayesian Quantized Neural Networks

Bayesian learning of model parameters in neural networks is important in...
research
12/15/2020

Scalable Verification of Quantized Neural Networks (Technical Report)

Formal verification of neural networks is an active topic of research, a...
research
05/03/2020

A Little Bit More: Bitplane-Wise Bit-Depth Recovery

Imaging sensors digitize incoming scene light at a dynamic range of 10–1...
research
02/10/2021

Impact of Bit Allocation Strategies on Machine Learning Performance in Rate Limited Systems

Intelligent entities such as self-driving vehicles, with their data bein...
research
12/16/2021

Approximation of functions with one-bit neural networks

This paper examines the approximation capabilities of coarsely quantized...

Please sign up or login with your details

Forgot password? Click here to reset