Log In Sign Up

Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains

by   Samira Abnar, et al.

In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models. ReStA is a variant of the popular representational similarity analysis (RSA) in cognitive neuroscience. While RSA can be used to compare representations in models, model components, and human brains, ReStA compares instances of the same model, while systematically varying single model parameter. Using ReStA, we study four recent and successful neural language models, and evaluate how sensitive their internal representations are to the amount of prior context. Using RSA, we perform a systematic study of how similar the representational spaces in the first and second (or higher) layers of these models are to each other and to patterns of activation in the human brain. Our results reveal surprisingly strong differences between language models, and give insights into where the deep linguistic processing, that integrates information over multiple sentences, is happening in these models. The combination of ReStA and RSA on models and brains allows us to start addressing the important question of what kind of linguistic processes we can hope to observe in fMRI brain imaging data. In particular, our results suggest that the data on story reading from Wehbe et al. (2014) contains a signal of shallow linguistic processing, but show no evidence on the more interesting deep linguistic processing.


page 4

page 5

page 7

page 9

page 13


Joint processing of linguistic properties in brains and language models

Language models have been shown to be very effective in predicting brain...

Does injecting linguistic structure into language models lead to better alignment with brain recordings?

Neuroscientists evaluate deep neural networks for natural language proce...

Geospatial distributions reflect rates of evolution of features of language

Different structural features of human language change at different rate...

The Grammar-Learning Trajectories of Neural Language Models

The learning trajectories of linguistic phenomena provide insight into t...

Pretraining on Non-linguistic Structure as a Tool for Analyzing Learning Bias in Language Models

We propose a novel methodology for analyzing the encoding of grammatical...

Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps

Neural Language Models (NLMs) have made tremendous advances during the l...

Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?

This work presents a detailed linguistic analysis into why larger Transf...