DeepAI AI Chat
Log In Sign Up

FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition

by   Xingchen Song, et al.

The recently proposed Conformer architecture which combines convolution with attention to capture both local and global dependencies has become the de facto backbone model for Automatic Speech Recognition (ASR). Inherited from the Natural Language Processing (NLP) tasks, the architecture takes Layer Normalization (LN) as a default normalization technique. However, through a series of systematic studies, we find that LN might take 10% of the inference time despite that it only contributes to 0.1% of the FLOPs. This motivates us to replace LN with other normalization techniques, e.g., Batch Normalization (BN), to speed up inference with the help of operator fusion methods and the avoidance of calculating the mean and variance statistics during inference. After examining several plain attempts which directly remove all LN layers or replace them with BN in the same place, we find that the divergence issue is mainly caused by the unstable layer output. We therefore propose to append a BN layer to each linear or convolution layer where stabilized training results are observed. We also propose to simplify the activations in Conformer, such as Swish and GLU, by replacing them with ReLU. All these exchanged modules can be fused into the weights of the adjacent linear/convolution layers and hence have zero inference cost. Therefore, we name it FusionFormer. Our experiments indicate that FusionFormer is as effective as the LN-based Conformer and is about 10% faster.


Squeezeformer: An Efficient Transformer for Automatic Speech Recognition

The recently proposed Conformer model has become the de facto backbone m...

Attention Enhanced Citrinet for Speech Recognition

Citrinet is an end-to-end convolutional Connectionist Temporal Classific...

Deep Sparse Conformer for Speech Recognition

Conformer has achieved impressive results in Automatic Speech Recognitio...

SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition

Very deep CNNs achieve state-of-the-art results in both computer vision ...

Light Gated Recurrent Units for Speech Recognition

A field that has directly benefited from the recent advances in deep lea...

Unified Normalization for Accelerating and Stabilizing Transformers

Solid results from Transformers have made them prevailing architectures ...