DeepAI AI Chat
Log In Sign Up

Attribution Analysis of Grammatical Dependencies in LSTMs

by   Yiding Hao, et al.
Yale University

LSTM language models have been shown to capture syntax-sensitive grammatical dependencies such as subject-verb agreement with a high degree of accuracy (Linzen et al., 2016, inter alia). However, questions remain regarding whether they do so using spurious correlations, or whether they are truly able to match verbs with their subjects. This paper argues for the latter hypothesis. Using layer-wise relevance propagation (Bach et al., 2015), a technique that quantifies the contributions of input features to model behavior, we show that LSTM performance on number agreement is directly correlated with the model's ability to distinguish subjects from other nouns. Our results suggest that LSTM language models are able to infer robust representations of syntactic dependencies.


Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State

We deploy the methods of controlled psycholinguistic experimentation to ...

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

The success of long short-term memory (LSTM) neural networks in language...

Do LSTMs See Gender? Probing the Ability of LSTMs to Learn Abstract Syntactic Rules

LSTMs trained on next-word prediction can accurately perform linguistic ...

Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

State-of-the-art LSTM language models trained on large corpora learn seq...

Assessing the Unitary RNN as an End-to-End Compositional Model of Syntax

We show that both an LSTM and a unitary-evolution recurrent neural netwo...

Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?

The main subject and the associated verb in English must agree in gramma...

Unsupervised inference approach to facial attractiveness

The perception of facial beauty is a complex phenomenon depending on man...