DeepAI AI Chat
Log In Sign Up

How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?

by   Gantavya Bhatt, et al.
Indian Institute of Technology Delhi

Long short-term memory (LSTM) networks and their variants are capable of encapsulating long-range dependencies, which is evident from their performance on a variety of linguistic tasks. On the other hand, simple recurrent networks (SRNs), which appear more biologically grounded in terms of synaptic connections, have generally been less successful at capturing long-range dependencies as well as the loci of grammatical errors in an unsupervised setting. In this paper, we seek to develop models that bridge the gap between biological plausibility and linguistic competence. We propose a new architecture, the Decay RNN, which incorporates the decaying nature of neuronal activations and models the excitatory and inhibitory connections in a population of neurons. Besides its biological inspiration, our model also shows competitive performance relative to LSTMs on subject-verb agreement, sentence grammaticality, and language modeling tasks. These results provide some pointers towards probing the nature of the inductive biases required for RNN architectures to model linguistic phenomena successfully.


Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

The success of long short-term memory (LSTM) neural networks in language...

Long-Short Range Context Neural Networks for Language Modeling

The goal of language modeling techniques is to capture the statistical a...

Exploring Processing of Nested Dependencies in Neural-Network Language Models and Humans

Recursive processing in sentence comprehension is considered a hallmark ...

Visualizing and Understanding Recurrent Networks

Recurrent Neural Networks (RNNs), and specifically a variant with Long S...

Learning in Competitive Network with Haeusslers Equation adapted using FIREFLY algorithm

Many of the competitive neural network consists of spatially arranged ne...

Learning long-term dependencies for action recognition with a biologically-inspired deep network

Despite a lot of research efforts devoted in recent years, how to effici...

Assessing the Unitary RNN as an End-to-End Compositional Model of Syntax

We show that both an LSTM and a unitary-evolution recurrent neural netwo...