DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks

07/25/2019
by   Lin Zehui, et al.
0

Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurrent and convolutional layers, the fully-connected self-attention layer surprisingly lacks a specific dropout method. This paper explores the possibility of regularizing the attention weights in Transformers to prevent different contextualized feature vectors from co-adaption. Experiments on a wide range of tasks show that DropAttention can improve performance and reduce overfitting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/14/2018

On the relationship between Dropout and Equiangular Tight Frames

Dropout is a popular regularization technique in neural networks. Yet, t...
research
07/26/2017

Reduction of Overfitting in Diabetes Prediction Using Deep Learning Neural Network

Augmented accuracy in prediction of diabetes will open up new frontiers ...
research
06/09/2023

Weight Freezing: A Regularization Approach for Fully Connected Layers with an Application in EEG Classification

In the realm of EEG decoding, enhancing the performance of artificial ne...
research
11/16/2018

DropFilter: A Novel Regularization Method for Learning Convolutional Neural Networks

The past few years have witnessed the fast development of different regu...
research
09/21/2021

Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruning

This paper is devoted to the features of the practical application of El...
research
11/11/2022

Multilevel-in-Layer Training for Deep Neural Network Regression

A common challenge in regression is that for many problems, the degrees ...
research
07/23/2020

WeightNet: Revisiting the Design Space of Weight Networks

We present a conceptually simple, flexible and effective framework for w...

Please sign up or login with your details

Forgot password? Click here to reset