Representation of linguistic form and function in recurrent neural networks

02/29/2016
by   Akos Kadar, et al.
0

We present novel methods for analyzing the activation patterns of RNNs from a linguistic point of view and explore the types of linguistic structure they learn. As a case study, we use a multi-task gated recurrent network architecture consisting of two parallel pathways with shared word embeddings trained on predicting the representations of the visual scene corresponding to an input sentence, and predicting the next word in the same sentence. Based on our proposed method to estimate the amount of contribution of individual tokens in the input to the final prediction of the networks we show that the image prediction pathway: a) is sensitive to the information structure of the sentence b) pays selective attention to lexical categories and grammatical functions that carry semantic information c) learns to treat the same input token differently depending on its grammatical functions in the sentence. In contrast the language model is comparatively more sensitive to words with a syntactic function. Furthermore, we propose methods to ex- plore the function of individual hidden units in RNNs and show that the two pathways of the architecture in our case study contain specialized units tuned to patterns informative for the task, some of which can carry activations to later time steps to encode long-term dependencies.

READ FULL TEXT

page 5

page 11

research
04/21/2018

Multi-task Learning for Universal Sentence Representations: What Syntactic and Semantic Information is Captured?

Learning distributed sentence representations is one of the key challeng...
research
04/23/2018

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model

We show how to deploy recurrent neural networks within a hierarchical Ba...
research
06/04/2022

Comparing Performance of Different Linguistically-Backed Word Embeddings for Cyberbullying Detection

In most cases, word embeddings are learned only from raw tokens or in so...
research
06/17/2019

Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text

Recurrent neural networks (RNNs) have reached striking performance in ma...
research
03/29/2018

Colorless green recurrent networks dream hierarchically

Recurrent neural networks (RNNs) have achieved impressive results in a v...
research
08/22/2019

RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

Recurrent neural networks (RNNs) are particularly well-suited for modeli...
research
03/18/2021

Phylogenetic typology

In this article we propose a novel method to estimate the frequency dist...

Please sign up or login with your details

Forgot password? Click here to reset