Rethinking the Value of Transformer Components

11/07/2020
by   Wenxuan Wang, et al.
0

Transformer becomes the state-of-the-art translation model, while it is not well studied how each intermediate component contributes to the model performance, which poses significant challenges for designing optimal architectures. In this work, we bridge this gap by evaluating the impact of individual component (sub-layer) in trained Transformer models from different perspectives. Experimental results across language pairs, training strategies, and model capacities show that certain components are consistently more important than the others. We also report a number of interesting findings that might help humans better analyze, understand and improve Transformer models. Based on these observations, we further propose a new training strategy that can improves translation performance by distinguishing the unimportant components in training.

READ FULL TEXT

page 3

page 4

page 5

page 6

page 7

page 8

research
03/18/2019

Neutron: An Implementation of the Transformer Translation Model and its Variants

The Transformer translation model is easier to parallelize and provides ...
research
05/03/2020

On the Inference Calibration of Neural Machine Translation

Confidence calibration, which aims to make model predictions equal to th...
research
08/30/2019

Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel

Transformer is a powerful architecture that achieves superior performanc...
research
09/12/2023

How does representation impact in-context learning: A exploration on a synthetic task

In-context learning, i.e., learning from in-context samples, is an impre...
research
02/02/2023

FCB-SwinV2 Transformer for Polyp Segmentation

Polyp segmentation within colonoscopy video frames using deep learning m...
research
09/15/2021

Sequence Length is a Domain: Length-based Overfitting in Transformer Models

Transformer-based sequence-to-sequence architectures, while achieving st...

Please sign up or login with your details

Forgot password? Click here to reset