Preventing RNN from Using Sequence Length as a Feature

12/16/2022
by   Jean-Thomas Baillargeon, et al.
0

Recurrent neural networks are deep learning topologies that can be trained to classify long documents. However, in our recent work, we found a critical problem with these cells: they can use the length differences between texts of different classes as a prominent classification feature. This has the effect of producing models that are brittle and fragile to concept drift, can provide misleading performances and are trivially explainable regardless of text content. This paper illustrates the problem using synthetic and real-world data and provides a simple solution using weight decay regularization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/30/2019

TS-RNN: Text Steganalysis Based on Recurrent Neural Networks

With the rapid development of natural language processing technologies, ...
research
04/09/2019

Seq2Biseq: Bidirectional Output-wise Recurrent Neural Networks for Sequence Modelling

During the last couple of years, Recurrent Neural Networks (RNN) have re...
research
01/09/2020

Internal representation dynamics and geometry in recurrent neural networks

The efficiency of recurrent neural networks (RNNs) in dealing with seque...
research
09/29/2021

On the Provable Generalization of Recurrent Neural Networks

Recurrent Neural Network (RNN) is a fundamental structure in deep learni...
research
09/28/2020

Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach

This paper is an attempt to bridge the gap between deep learning and gra...
research
11/12/2015

Improving performance of recurrent neural network with relu nonlinearity

In recent years significant progress has been made in successfully train...
research
12/16/2022

Reducing Sequence Length Learning Impacts on Transformer Models

Classification algorithms using Transformer architectures can be affecte...

Please sign up or login with your details

Forgot password? Click here to reset