Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition

06/30/2022

∙

We present a novel sub-8-bit quantization-aware training (S8BQAT) scheme for 8-bit neural network accelerators. Our method is inspired from Lloyd-Max compression theory with practical adaptations for a feasible computational overhead during training. With the quantization centroids derived from a 32-bit baseline, we augment training loss with a Multi-Regional Absolute Cosine (MRACos) regularizer that aggregates weights towards their nearest centroid, effectively acting as a pseudo compressor. Additionally, a periodically invoked hard compressor is introduced to improve the convergence rate by emulating runtime model weight quantization. We apply S8BQAT on speech recognition tasks using Recurrent Neural NetworkTransducer (RNN-T) architecture. With S8BQAT, we are able to increase the model parameter size to reduce the word error rate by 4-16

READ FULL TEXT

Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

Accelerator-Aware Training for Transducer-Based Speech Recognition

RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models

Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets

DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization

Accelerating Inference and Language Model Fusion of Recurrent Neural Network Transducers via End-to-End 4-bit Quantization

Multi-objective Recurrent Neural Networks Optimization for the Edge – a Quantization-based Approach

Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition

Related Research

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

Accelerator-Aware Training for Transducer-Based Speech Recognition

RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models

Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets

DEEPEYE: A Compact and Accurate Video Comprehension at Terminal Devices Compressed with Quantization and Tensorization

Accelerating Inference and Language Model Fusion of Recurrent Neural Network Transducers via End-to-End 4-bit Quantization

Multi-objective Recurrent Neural Networks Optimization for the Edge – a Quantization-based Approach