Convolutional Self-Attention Network

10/31/2018
by   Baosong Yang, et al.
0

Self-attention network (SAN) has recently attracted increasing interest due to its fully parallelized computation and flexibility in modeling dependencies. It can be further enhanced with multi-headed attention mechanism by allowing the model to jointly attend to information from different representation subspaces at different positions (Vaswani et al., 2017). In this work, we propose a novel convolutional self-attention network (CSAN), which offers SAN the abilities to 1) capture neighboring dependencies, and 2) model the interaction between multiple attention heads. Experimental results on WMT14 English-to-German translation task demonstrate that the proposed approach outperforms both the strong Transformer baseline and other existing works on enhancing the locality of SAN. Comparing with previous work, our model does not introduce any new parameters.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/05/2019

Convolutional Self-Attention Networks

Self-attention networks (SANs) have drawn increasing interest due to the...
research
11/23/2021

Boosting Neural Machine Translation with Dependency-Scaled Self-Attention Network

The neural machine translation model assumes that syntax knowledge can b...
research
10/24/2018

Modeling Localness for Self-Attention Networks

Self-attention networks have proven to be of profound value for its stre...
research
09/15/2020

Cascaded Semantic and Positional Self-Attention Network for Document Classification

Transformers have shown great success in learning representations for la...
research
12/29/2020

Multiple Structural Priors Guided Self Attention Network for Language Understanding

Self attention networks (SANs) have been widely utilized in recent NLP s...
research
05/18/2021

Progressively Normalized Self-Attention Network for Video Polyp Segmentation

Existing video polyp segmentation (VPS) models typically employ convolut...
research
10/07/2021

Using Keypoint Matching and Interactive Self Attention Network to verify Retail POSMs

Point of Sale Materials(POSM) are the merchandising and decoration items...

Please sign up or login with your details

Forgot password? Click here to reset