Benchmarking the Reliability of Post-training Quantization: a Particular Focus on Worst-case Performance

03/23/2023
by   Zhihang Yuan, et al.
0

Post-training quantization (PTQ) is a popular method for compressing deep neural networks (DNNs) without modifying their original architecture or training procedures. Despite its effectiveness and convenience, the reliability of PTQ methods in the presence of some extrem cases such as distribution shift and data noise remains largely unexplored. This paper first investigates this problem on various commonly-used PTQ methods. We aim to answer several research questions related to the influence of calibration set distribution variations, calibration paradigm selection, and data augmentation or sampling strategies on PTQ reliability. A systematic evaluation process is conducted across a wide range of tasks and commonly-used PTQ paradigms. The results show that most existing PTQ methods are not reliable enough in term of the worst-case group performance, highlighting the need for more robust methods. Our findings provide insights for developing PTQ methods that can effectively handle distribution shift scenarios and enable the deployment of quantized DNNs in real-world applications.

READ FULL TEXT

page 1

page 5

page 6

page 15

page 16

page 17

research
04/08/2023

Benchmarking the Robustness of Quantized Models

Quantization has emerged as an essential technique for deploying deep ne...
research
08/04/2023

RobustMQ: Benchmarking Robustness of Quantized Models

Quantization has emerged as an essential technique for deploying deep ne...
research
06/07/2019

Fighting Quantization Bias With Bias

Low-precision representation of deep neural networks (DNNs) is critical ...
research
11/25/2019

A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift

The uncertainty estimation is critical in real-world decision making app...
research
06/30/2023

Designing strong baselines for ternary neural network quantization through support and mass equalization

Deep neural networks (DNNs) offer the highest performance in a wide rang...
research
03/25/2023

Towards Accurate Post-Training Quantization for Vision Transformer

Vision transformer emerges as a potential architecture for vision tasks....
research
06/30/2023

Impact of Noise on Calibration and Generalisation of Neural Networks

Noise injection and data augmentation strategies have been effective for...

Please sign up or login with your details

Forgot password? Click here to reset