Evaluating the Ability of LSTMs to Learn Context-Free Grammars

11/06/2018
by   Luzi Sennhauser, et al.
0

While long short-term memory (LSTM) neural net architectures are designed to capture sequence information, human language is generally composed of hierarchical structures. This raises the question as to whether LSTMs can learn hierarchical structures. We explore this question with a well-formed bracket prediction task using two types of brackets modeled by an LSTM. Demonstrating that such a system is learnable by an LSTM is the first step in demonstrating that the entire class of CFLs is also learnable. We observe that the model requires exponential memory in terms of the number of characters and embedded depth, where a sub-linear memory should suffice. Still, the model does more than memorize the training input. It learns how to distinguish between relevant and irrelevant information. On the other hand, we also observe that the model does not generalize well. We conclude that LSTMs do not learn the relevant underlying context-free rules, suggesting the good overall performance is attained rather by an efficient way of evaluating nuisance variables. LSTMs are a way to quickly reach good results for many natural language tasks, but to understand and generate natural language one has to investigate other concepts that can make more direct use of natural language's structural nature.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2018

LSTMs Exploit Linguistic Attributes of Data

While recurrent neural networks have found success in a variety of natur...
research
11/04/2016

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

The success of long short-term memory (LSTM) neural networks in language...
research
05/23/2018

Implicit Language Model in LSTM for OCR

Neural networks have become the technique of choice for OCR, but many as...
research
10/25/2019

A memory enhanced LSTM for modeling complex temporal dependencies

In this paper, we present Gamma-LSTM, an enhanced long short term memory...
research
11/15/2017

Variational Bi-LSTMs

Recurrent neural networks like long short-term memory (LSTM) are importa...
research
08/26/2019

Using LSTMs to Model the Java Programming Language

Recurrent neural networks (RNNs), specifically long-short term memory ne...
research
08/07/2015

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Natural language generation (NLG) is a critical component of spoken dial...

Please sign up or login with your details

Forgot password? Click here to reset