DeepAI AI Chat
Log In Sign Up

RNNs can generate bounded hierarchical languages with optimal memory

10/15/2020
by   John Hewitt, et al.
23

Recurrent neural networks empirically generate natural language with high syntactic fidelity. However, their success is not well-understood theoretically. We provide theoretical insight into this success, proving in a finite-precision setting that RNNs can efficiently generate bounded hierarchical languages that reflect the scaffolding of natural language syntax. We introduce Dyck-(k,m), the language of well-nested brackets (of k types) and m-bounded nesting depth, reflecting the bounded memory needs and long-distance dependencies of natural language syntax. The best known results use O(k^m/2) memory (hidden units) to generate these languages. We prove that an RNN with O(m log k) hidden units suffices, an exponential reduction in memory, by an explicit construction. Finally, we show that no algorithm, even with unbounded computation, can suffice with o(m log k) hidden units.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/24/2021

Self-Attention Networks Can Process Bounded Hierarchical Languages

Despite their impressive performance in NLP, self-attention networks wer...
11/08/2020

On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages

While recurrent models have been effective in NLP tasks, their performan...
04/25/2023

Nondeterministic Stacks in Neural Networks

Human language is full of compositional syntactic structures, and althou...
02/18/2020

Assessing the Memory Ability of Recurrent Neural Networks

It is known that Recurrent Neural Networks (RNNs) can remember, in their...
03/23/2020

Depth Enables Long-Term Memory for Recurrent Neural Networks

A key attribute that drives the unprecedented success of modern Recurren...
09/05/2018

RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency

Recurrent neural networks (RNNs) are the state of the art in sequence mo...
06/04/2019

Sequential Neural Networks as Automata

This work attempts to explain the types of computation that neural netwo...