LSTMs Exploit Linguistic Attributes of Data

05/29/2018
by   Nelson F. Liu, et al.
0

While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/06/2018

Evaluating the Ability of LSTMs to Learn Context-Free Grammars

While long short-term memory (LSTM) neural net architectures are designe...
research
04/30/2020

Pretraining on Non-linguistic Structure as a Tool for Analyzing Learning Bias in Language Models

We propose a novel methodology for analyzing the encoding of grammatical...
research
05/03/2020

Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models

LSTM-based recurrent neural networks are the state-of-the-art for many n...
research
02/15/2021

Certified Robustness to Programmable Transformations in LSTMs

Deep neural networks for natural language processing are fragile in the ...
research
04/22/2021

Finding Fuzziness in Neural Network Models of Language Processing

Humans often communicate by using imprecise language, suggesting that fu...
research
08/31/2018

Neural DrugNet

In this paper, we describe the system submitted for the shared task on S...
research
11/10/2019

Location Attention for Extrapolation to Longer Sequences

Neural networks are surprisingly good at interpolating and perform remar...

Please sign up or login with your details

Forgot password? Click here to reset