LSTMs Compose (and Learn) Bottom-Up

10/06/2020
by   Naomi Saphra, et al.
0

Recent work in NLP shows that LSTM language models capture hierarchical structure in language data. In contrast to existing work, we consider the learning process that leads to their compositional behavior. For a closer look at how an LSTM's sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We connect this measure to syntax with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hierarchical structures are discovered over the course of training: that LSTM constituent representations are learned bottom-up, relying on effective representations of their shorter children, rather than learning the longer-range relations independently from children.

READ FULL TEXT
research
04/27/2020

Word Interdependence Exposes How LSTMs Compose Representations

Recent work in NLP shows that LSTM language models capture compositional...
research
09/27/2020

Multi-timescale representation learning in LSTM Language Models

Although neural language models are effective at capturing statistics of...
research
03/18/2019

The emergence of number and syntax units in LSTM language models

Recent work has shown that LSTMs trained on a generic language modeling ...
research
05/14/2018

Word learning and the acquisition of syntactic--semantic overhypotheses

Children learning their first language face multiple problems of inducti...
research
06/26/2020

What they do when in doubt: a study of inductive biases in seq2seq learners

Sequence-to-sequence (seq2seq) learners are widely used, but we still ha...
research
06/08/2022

Syntactic Inductive Biases for Deep Learning Methods

In this thesis, we try to build a connection between the two schools by ...
research
11/27/2016

The polysemy of the words that children learn over time

Here we study polysemy as a potential learning bias in vocabulary learni...

Please sign up or login with your details

Forgot password? Click here to reset