DeepAI AI Chat
Log In Sign Up

On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

by   Tianchu Ji, et al.

How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accuracies. However, this requires retraining or even creating entirely new models, both of which can be expensive and carbon-emitting. Focused on optimizations that do not require training, we systematically study the full range of typical attention values necessary. This informs the design of an inference-time quantization technique using both pruning and log-scaled mapping which produces only a few (e.g. 2^3) unique values. Over the tasks of question answering and sentiment analysis, we find nearly 80 zeros with minimal (< 1.0%) relative loss in accuracy. We use this pruning technique in conjunction with quantizing the attention values to only a 3-bit format, without retraining, resulting in only a 0.8 question answering with fine-tuned RoBERTa.


Pruning a BERT-based Question Answering Model

We investigate compressing a BERT-based question answering system by pru...

DoT: An efficient Double Transformer for NLP tasks with tables

Transformer-based approaches have been successfully used to obtain state...

Optimizing Inference Performance of Transformers on CPUs

The Transformer architecture revolutionized the field of natural languag...

A2P-MANN: Adaptive Attention Inference Hops Pruned Memory-Augmented Neural Networks

In this work, to limit the number of required attention inference hops i...

idT5: Indonesian Version of Multilingual T5 Transformer

Indonesian language is spoken by almost 200 million people and is the 10...

MEMO : Accelerating Transformers with Memoization on Big Memory Systems

Transformers gain popularity because of their superior prediction accura...

Zero-Shot Dynamic Quantization for Transformer Inference

We introduce a novel run-time method for significantly reducing the accu...