Linformer: Self-Attention with Linear Complexity

by   Sinong Wang, et al.

Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses O(n^2) time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from O(n^2) to O(n) in both time and space. The resulting linear transformer, the Linformer, performs on par with standard Transformer models, while being much more memory- and time-efficient.


page 1

page 2

page 3

page 4


You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling

Transformer-based models are widely used in natural language processing ...

IS Attention All What You Need? – An Empirical Investigation on Convolution-Based Active Memory and Self-Attention

The key to a Transformer model is the self-attention mechanism, which al...

On The Computational Complexity of Self-Attention

Transformer architectures have led to remarkable progress in many state-...

A Tensorized Transformer for Language Modeling

Latest development of neural models has connected the encoder and decode...

Adaptive Multi-Resolution Attention with Linear Complexity

Transformers have improved the state-of-the-art across numerous tasks in...

Sequential Recommendation with Relation-Aware Kernelized Self-Attention

Recent studies identified that sequential Recommendation is improved by ...

Temporal Attention for Language Models

Pretrained language models based on the transformer architecture have sh...

Code Repositories


My take on a practical implementation of Linformer for Pytorch.

view repo


Reproducing the Linear Multihead Attention introduced in Linformer paper (Linformer: Self-Attention with Linear Complexity)

view repo


Sparse Transformer with limited attention span in PyTorch

view repo

1 Introduction

Transformer models (vaswani2017attention) have become ubiquitous for wide variety of problems in natural language processing (NLP), including translation (ott2018scaling), text classification, question answering, among others (raffel2019exploring; mohamed2019transformers). Over the last couple of years, the number of parameters in state-of-the-art NLP transformers has grown drastically, from the original 340 million introduced in BERT-Large to 175 billion in GPT-3 (brown2020language). Although these large-scale models yield impressive results on wide variety of tasks, training and deploying such model are slow in practice. For example, the original BERT-Large model (devlin2019bert) takes four days to train on 16 Cloud TPUs, and the recent GPT-3 (brown2020language)

consumed orders of magnitude more petaflops / day to train compared to its predecessor, GPT-2 

(radford2019language). Beyond training, deploying Transformer models to real world applications is also expensive, usually requiring extensive distillation (hinton2015distilling) or compression.

The main efficiency bottleneck in Transformer models is its self-attention mechanism. Here, each token’s representation is updated by attending to all other tokens in the previous layer. This operation is key for retaining long-term information, giving Transformers the edge over recurrent models on long sequences. However, attending to all tokens at each layer incurs a complexity of with respect to sequence length. Thus, in this paper, we seek to answer the question: can Transformer models be optimized to avoid this quadratic operation, or is this operation required to maintain strong performance?

Prior work has proposed several techniques for improving the efficiency of self-attention. One popular technique is introducing sparsity into attention layers (child2019generating; qiu2019blockwise; beltagy2020longformer) by having each token attend to only a subset of tokens in the whole sequence. This reduces the overall complexity of the attention mechanism to  (child2019generating). However, as shown in qiu2019blockwise, this approach suffers from a large performance drop with limited efficiency gains, i.e., a 2% drop with only 20% speed up. More recently, the Reformer (kitaev2019reformer) used locally-sensitive hashing (LSH) to reduce the self-attention complexity to . However, in practice, the Reformer’s efficiency gains only appear on sequences with length (Figure 5 in kitaev2019reformer). Furthermore, the Reformer’s multi-round hashing approach actually increases the number of sequential operations, which further undermines their final efficiency gains.

In this work, we introduce a novel approach for tackling the self-attention bottleneck in Transformers. Our approach is inspired by the key observation that self-attention is low rank

. More precisely, we show both theoretically and empirically that the stochastic matrix formed by self-attention can be approximated by a low-rank matrix. Empowered by this observation, we introduce a novel mechanism that reduces self-attention to an

operation in both space- and time-complexity: we decompose the original scaled dot-product attention into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention. A summary of runtimes for various Transformer architectures, including ours, can be found in Table 1.

One predominant application of Transformers, that has seen the most gains, is using them as pretrained language models, whereby models are first pretrained with a language modeling objective on a large corpus, then finetuned on target tasks using supervised data (devlin2019bert; liu2019roberta; lewis2019bart). Following devlin2019bert, we pretrain our model on BookCorpus (zhu2015aligning) plus English Wikipedia using masked-language-modeling objective. We observe similar pretraining performance to the standard Transformer model. We then finetune our pretrained models on three tasks from GLUE (DBLP:journals/corr/abs-1804-07461)

and one sentiment analysis task, IMDB reviews 

(maas2011learning). On these tasks, we find that our model performs comparably, or even slightly better, than the standard pretrained Transformer, while observing significant training and inference speedups.

Model Architecture Complexity per Layer Sequential Operation
Transformer, (vaswani2017attention)
Sparse Tansformer, (child2019generating)
Reformer, (kitaev2019reformer)
Table 1: Per-layer time complexity and minimum number of sequential operations as a function of sequence length () for various architectures.

2 Backgrounds and Related works

2.1 Transformer and Self-Attention

The Transformer is built upon the idea of Multi-Head Self-Attention (MHA), which allows the model to jointly attend to information at different positions from different representation subspaces. MHA is defined as


where are input embedding matrices, is sequence length, is the embedding dimension, and is the number of heads. Each head is defined as:


where are learned matrices and are the hidden dimensions of the projection subspaces. For the rest of this paper, we will not differentiate between and and just use .

The self-attention defined in (2) refers to a context mapping matrix . The Transformer uses to capture the input context for a given token, based on a combination of all tokens in the sequence. However, computing is expensive. It requires multiplying two matrices, which is in time and space complexity. This quadratic dependency on the sequence length has become a bottleneck for Transformers.

2.2 Related works

There has been much prior literature on improving the efficiency of Transformers, especially the self-attention bottleneck. The most common techniques for model efficiency that can be applied to Transformers (some specific to Transformers, others more general-purpose) include:

Mixed Precision (micikevicius2017mixed)

: Using half-precision or mixed-precision representations of floating points is popular in deep learning, and is also widely used in training Transformers 

(ott2019fairseq). This technique can be further improved through Quantization Aware Training (jacob2018quantization; fan2020training)

, where the weights are quantized during training and the gradients are approximated with the Straight-Through Estimator. This line of work is orthogonal to our approach, and we use mixed-precision training by default.

Knowledge Distillation (hinton2015distilling): Knowledge distillation aims to transfer the “knowledge" from a large teacher model to a lightweight student model. The student model is then used during inference. However this approach has drawbacks: It does not address speeding up the teacher model during training, and moreover, student models usually suffer performance degradation compared to the teacher model. For example, when distilling a 12-layer BERT to a 6-layer BERT, the student model experiences an average 2.5% performance drop on several benchmark tasks (sanh2019distilbert).

Sparse Attention (child2019generating): This technique improves the efficiency of self-attention by adding sparsity in the context mapping matrix . For example, the Sparse Transformer (child2019generating) only computes around the diagonal of matrix (instead of the all ). Meanwhile, blockwise self-attention (qiu2019blockwise) divides into multiple blocks and only computes within the selected blocks. However, these techniques also suffer a large performance degradation, while having only limited additional speed-up, i.e., 2% drop with 20% speed up.

LSH Attention (kitaev2019reformer): Locally-sensitive hashing (LSH) attention utilizes a multi-round hashing scheme when computing dot-product attention, which in theory reduces the self-attention complexity to . However, in practice, their complexity term has a large constant and it is only more efficient than the vanilla transformer when sequence length is extremely long.

Improving Optimizer Efficiency: Microbatching (huang2019gpipe) splits a batch into small microbatches (which can be fit into memory), and then separately runs forward and backward passes on them with gradient accumulation. Gradient checkpointing (chen2016training)

saves memory by only caching activations of a subset of layers. The uncached activations are recomputed during backpropagation from the latest checkpoint. Both techniques trade off time for memory, and do not speed up inference.

As we’ve noted, most common techniques have limitations in reducing both the training and inference time/memory consumption, we investigate how to optimize the self-attention layers and introduce our approach next.

3 Self-Attention is Low Rank

In this section, we demonstrate that the self-attention mechanism, i.e., the context mapping matrix , is low-rank.

Figure 1: Left two figures are spectrum analysis of the self-attention matrix in pretrained transformer model (liu2019roberta) with

. The Y-axis is the normalized cumulative singular value of context mapping matrix

, and the X-axis the index of largest eigenvalue. The results are based on both RoBERTa-base and large model in two public datasets: Wiki103 and IMDB. The right figure plots the heatmap of normalized cumulative eigenvalue at the 128-th largest eigenvalue across different layers and heads in Wiki103 data.

We first provide a spectrum analysis of the context mapping matrix . We use two pretrained transformer models, RoBERTa-base (12-layer stacked transformer) and RoBERTa-large (24-layer stacked transformer) (liu2019roberta) on two tasks: masked-language-modeling task on Wiki103 (merity2016pointer) and classification task on IMDB (maas2011learning). In Figure 1

(left), we apply singular value decomposition into

across different layers and different heads of the model, and plot the normalized cumulative singular value averaged over 10k sentences. The results exhibit a clear long-tail spectrum distribution across each layer, head and task. This implies that most of the information of matrix can be recovered from the first few largest singular values. In Figure 1

(right), we plot a heatmap of the normalized cumulative singular value at the 128-th largest singular value (out of 512). We observe that the spectrum distribution in higher layers is more skewed than in lower layers, meaning that, in higher layers, more information is concentrated in the largest singular values and the rank of

is lower.

Below, we provide a theoretical analysis of the above spectrum results.

Theorem 1.

(self-attention is low rank) For any and

, for any column vector

of matrix , there exists a low-rank matrix such that


where the context mapping matrix is defined in (2).


Based on the definition of the context mapping matrix , we can write


where is an diagonal matrix. The main idea of this proof is based on the distributional Johnson–Lindenstrauss lemma (lindenstrauss1984extensions) (JL for short). We construct the approximate low rank matrix as , where with i.i.d. entries from . We can then use the JL lemma to show that, for any column vector of matrix , when , we have


For more details, refer to the supplementary materials. ∎

Given the low-rank property of the context mapping matrix , one straightforward idea is to use singular value decomposition (SVD) to approximate with a low-rank matrix , as follows


where , and are the largest singular values and their corresponding singular vectors. Based on the results in Theorem 1 and the Eckart–Young–Mirsky Theorem (eckart1936approximation), one can use to approximate self-attention (2) with error and time and space complexity. However, this approach requires performing an SVD decomposition in each self-attention matrix, which adds additional complexity. Therefore, we propose another approach for low-rank approximation that avoids this added complexity.

4 Model

In this section, we propose a new self-attention mechanism which allows us to compute the contextual mapping in linear time and memory complexity with respect to sequence length.

The main idea of our proposed linear self-attention (Figure 2) is to add two linear projection matrices when computing key and value. We first project the original -dimensional key and value layers and into -dimensional projected key and value layers. We then compute an -dimensional context mapping matrix using scaled dot-product attention.


Finally, we compute context embeddings for each head using . Note the above operations only require time and space complexity. Thus, if we can choose a very small projected dimension , such that , then we can significantly reduce the memory and space consumption. The following theorem states that, when (independent of ), one can approximate using linear self-attention (7) with error.

Figure 2: Left and bottom-right show architecture and example of our proposed multihead linear self-attention. Top right shows inference time vs. sequence length for various Linformer models.
Theorem 2.

(Linear self-attention) For any and , if , then there exists matrices such that, for any row vector of matrix , we have


The main idea of proof is based on the distributional Johnson–Lindenstrauss lemma (lindenstrauss1984extensions). We first prove that for any row vector of matrix and column vector of matrix ,


where and , where with i.i.d. entries from and is a small constant. Applying the result in (9) to every row vector of matrix and every column vector of matrix , one can directly prove that, for any row vector of matrix ,


by setting . This result does not utilize the low rank property of matrix (rank()=) and the resultant has a dependency on sequence length . We will further utlize the fact that rank()= to prove the choice of can be constant and independent of sequence length . For more details, refer to the supplementary materials. ∎

In Figure 2 (top right), we plot the inference speed of Linformer and standard Transformer versus sequence length, while holding the total number of tokens fixed. We see that while standard Transformer becomes slower at longer sequence lengths, the Linformer speed remains relatively flat and is significantly faster at long sequences.

Additional Efficiency Techniques

Several additional techniques can be introduced on top of Linformer to further optimize for both performance and efficiency:

Parameter sharing between projections: One can share parameters for the linear projection matrices across layers and heads. In particular, we experimented with 3 levels of sharing:

  • Headwise sharing: for each layer, we share two projection matrices and such that and across all heads .

  • Key-value sharing: we do headwise sharing, with the additional constraint of sharing the key and value projections. For each layer, we create a single projection matrix such that for each key-value projection matrix across all head .

  • Layerwise sharing: we use a single projection matrix across all layers, for all heads, and for both key and value.

For example, in a 12-layer, 12-head stacked Transformer model, headwise sharing, key-value sharing and layerwise sharing will introduce 24, 12, and 1 distinct linear projection matrices, respectively.

Nonuniform projected dimension: One can choose a different projected dimension for different heads and layers. As shown in Figure 1 (right), the contextual mapping matrices in different heads and layers have distinct spectrum distributions, and heads in higher layer tend towards a more skewed distributed spectrum (lower rank). This implies one can choose a smaller projected dimension for higher layers.

General projections:

One can also choose different kinds of low-dimensional projection methods instead of a simple linear projection. For example, one can choose mean/max pooling, or convolution where the kernel and stride is set to

. The convolutional functions contain parameters that require training.

5 Experiments

In this section, we present experimental results for the the techniques described above. We analyze the techniques one-by-one and explore how they impact performance.

5.1 Pretraining Perplexities

We first compare the pretraining performance of our proposed architecture against RoBERTa (liu2019roberta), which is based on the Transformer. Following devlin2019bert, we use BookCorpus (zhu2015aligning) plus English Wikipedia as our pretraining set (3300M words). All models are pretrained with the masked-language-modeling (MLM) objective, and the training for all experiments are parallelized across 64 Tesla V100 GPUs with 250k updates.

Effect of projected dimension: We experiment with various values for the projected dimension . (We use the same across all layers and heads of Linformer.) In the Figure 3(a) and (b), we plot the validation perplexity curves for both the standard Transformer and the Linformer across different , for maximum sequence lengths and . As expected, the Linformer performs better as projected dimension increases. However, even at for and for , Linformer’s performance is already nearly on par with the original Transformer.

Figure 3: Pretraining validation perplexity versus number of updates.

Effect of sharing projections: In Figure 3(c), we plot the validation perplexity curves for the three parameter sharing strategies (headwise, key-value, and layerwise) with . Note that when we use just a single projection matrix (i.e. for layerwise sharing), the resulting Linformer model’s validation perplexity almost matches that of the the non-shared model. This suggests that we can decrease the number of additional parameters in our model, and consequently, it’s memory consumption, without much detriment to performance.

Effect of longer sequences: We evaluate the effect of sequence length during Linformer pretraining. In the Figure 3(d), we plot the validation perplexity for Linformer with , holding projected dimension fixed at . Note that as sequence length increases, even though our projected dimension is fixed, the final perplexities after convergence remain about the same. This further empirically supports our assertion that the Linformer is linear-time.

Model SST-2 IMDB QNLI QQP Average
512 liu2019roberta, RoBERTa-base 93.1 94.1 90.9 90.9 92.25
Linformer, 128 92.4 94.0 90.4 90.2 91.75
Linformer, 128, shared kv 93.4 93.4 90.3 90.3 91.85
Linformer, 128, shared kv, layer 93.2 93.8 90.1 90.2 91.83
Linformer, 256 93.2 94.0 90.6 90.5 92.08
Linformer, 256, shared kv 93.3 93.6 90.6 90.6 92.03
Linformer, 256, shared kv, layer 93.1 94.1 91.2 90.8 92.30
512 devlin2019bert, BERT-base 92.7 93.5 91.8 89.6 91.90
sanh2019distilbert, Distilled BERT 91.3 92.8 89.2 88.5 90.45
1024 Linformer, 256 93.0 93.8 90.4 90.4 91.90
Linformer, 256, shared kv 93.0 93.6 90.3 90.4 91.83
Linformer, 256, shared kv, layer 93.2 94.2 90.8 90.5 92.18
Table 2: Dev set results on benchmark natural language understanding tasks. The RoBERTa-base model here is pretrained with same corpus as BERT.

5.2 Downstream Results

Thus far, we have only examined the pretraining perplexities of our model. However, we wish to show that our conclusions hold after finetuning on downstream tasks. We finetune our Linformer on IMDB (maas2011learning) and SST-2 (socher2013recursive) (sentiment classification), as well as QNLI (natural language inference) (rajpurkar2016squad), and QQP (textual similarity) (chen2018quora) We do the same with RoBERTa, 12-layer BERT-base and 6-layer distilled BERT. All of our models, including the Transformer baselines, were pretrained with the same objective, pretraining corpus, and up to 250k updates (although our Linformer takes much less wall-clock time to get to 250k updates, and was consequently trained for less time). Results are listed in Table 2.

We observe that the Linformer model () has comparable downstream performance to the RoBERTa model, and in fact even slightly outperforms it at . Moreover, we note that although the Linformer’s layerwise sharing strategy shares a single projection matrix across the entire model, it actually exhibits the best accuracy result of all three parameter sharing strategies. Furthermore, the Linformer pretrained with longer sequence length has similar results to the one pretrained with shorter length , this empirically supports the notion that the performance of Linformer model is mainly determined by the projected dimension instead of the ratio .

5.3 Inference-time Efficiency Results

In Table 3, we report the inference efficiencies of Linformer (with layerwise sharing) against a standard Transformer. We benchmark both models’ inference speed and memory on a 16GB Tesla V100 GPU card. We randomly generate data up to some sequence length and perform a full forward pass on a multiple batches. We also choose batch size based on the maximum batch size that can fit in memory, and our memory savings are computed based on this number.

 length projected dimensions 128 256 512 1024 2048  512 1.5x 1.3x - - -  1024 1.7x 1.6x 1.3x - -  2048 2.6x 2.4x 2.1x 1.3x -  4096 3.4x 3.2x 2.8x 2.2x 1.3x  8192 5.5x 5.0x 4.4x 3.5x 2.1x  16384 8.6x 7.8x 7.0x 5.6x 3.3x  32768 13x 12x 11x 8.8x 5.0x  65536 20x 18x 16x 14x 7.9x  length projected dimensions 128 256 512 1024 2048  512 1.7x 1.5x - - -  1024 3.0x 2.9x 1.8x - -  2048 6.1x 5.6x 3.6x 2.0x -  4096 14x 13x 8.3x 4.3x 2.3x  8192 28x 26x 17x 8.5x 4.5x  16384 56x 48x 32x 16x 8x  32768 56x 48x 36x 18x 16x  65536 60x 52x 40x 20x 18x
Table 3: Inference-time efficiency improvements of the Linformer over the Transformer, across various projected dimensions and sequence lengths . Left table shows time saved. Right table shows memory saved.

From Table 3, we see that even with and , Linformer has faster inference time and allows for a larger maximum batch size than the Transformer. As sequence length increases, the inference-time speed-up and memory savings are even more dramatic. We also plot inference times of both Linformer and Transformer on the 100 data samples in the top right of Figure 2.

6 Conclusion

Transformer models are notoriously slow to train and deploy in practice since their self-attention operations have time and space complexity with respect to sequence length . In this paper, we demonstrate, both theoretically and empirically, that the stochastic matrix formed by self-attention mechanism is low-rank. We further leverage this observation to propose a new, highly efficient self-attention mechanism. Through a combination of theoretical and empirical analysis, we demonstrate that our proposed approach is with respect to sequence length.

Broader Impact

Our work focuses on making Transformers more efficient by introducing a mechanism that reduces self-attention to linear-time complexity. Potential positive impacts of efficient transformers include increasing the accessibility of our models, both for deployment on devices, as well as during training for research purposes. It also has potential impact on training transformer on images since we can support very long sequences. Furthermore, there are positive environmental benefits associated with decreasing the power consumption of models. As such, we see no immediate negative ethical or societal impacts of our work beyond what applies to other core building blocks of deep learning.


Appendix A Proof of Theorem 1


The main proof idea is based on the distributional Johnson–Lindenstrauss lemma (lindenstrauss1984extensions) (JL, for short), the following version is from (arriaga2006algorithmic).

Lemma 1.

Let be an matrix, , with i.i.d. entries from . For any , we have


For simplicity, we will omit the subscript for matrix , , , and . We will regard as , as and as . Define


Based on the definition of contextual mapping matrix , we have


where is an diagonal matrix such that


Here we provide a constructive proof. Given any approximation error , define the following matrix.


where be an matrix, , with i.i.d. entries from . Clearly the rank of matrix satisifies


We further show that, when , we have that, for any column vector ,


This concludes the theorem. For any row vector of matrix and any column vector of matrix , applying the JL Lemma, we can obtain


Therefore, we have


The above, step (a) is based on the union bound. The step (b) is utilizing the result of JL Lemma. Let , then theorem follows. ∎

Appendix B Proof of Theorem 2


Define and , where with i.i.d. entries from , is a constant with . We will first prove that for any row vector of matrix and column vector of matrix ,


Based on the triangle inequality, we have


The above, step (a) is based on the Cauchy inequality and JL Lemma in (11). The step (b) utilizes the fact that exponential function is Lipchitz continuous in a compact region. Then we can choose a small enough , i.e., such that


The step (c) is based on the JL Lemma defined in (12).

Applying the result in (21) to every row vector of matrix and every column vector of matrix , one can directly prove that, for any row vector of matrix ,


by setting . This result does not utilize the low rank property of matrix (rank()=) and the resultant has a dependency on sequence length . We will further prove the choice of can be constant and independent of sequence length .

Based on the fact that rank()=, we can find a row submatrix of matrix such that rank()=. Applying the result in (21) to every row vector of matrix and every column vector of matrix , and , we can obtain that, for any row vector of matrix ,


Furthermore, define the matrix as


We have that, for any row vector of matrix , .

The above, step (a) utilizes the inequality , where ( is the largest eigenvalue) is the spectrum norm of a matrix . The step (b) is based on matrix norm inequality , where is the Frobenius norm of matrix . The step (c) is based on the results of (24). ∎