Linking artificial and human neural representations of language

10/02/2019
by   Jon Gauthier, et al.
0

What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance. We find that none of the sentence encoding tasks tested yield significant increases in brain decoding performance. Through further task ablations and representational analyses, we find that tasks which produce syntax-light representations yield significant improvements in brain decoding performance. Our results constrain the space of NLU models that could best account for human neural representations of language, but also suggest limits on the possibility of decoding fine-grained syntactic information from fMRI human neuroimaging.

READ FULL TEXT
research
05/19/2023

Scaling laws for language encoding models in fMRI

Representations from transformer-based unidirectional language models ar...
research
06/02/2018

Does the brain represent words? An evaluation of brain decoding studies of language understanding

Language decoding studies have identified word representations which can...
research
06/27/2019

Relating Simple Sentence Representations in Deep Neural Networks and the Brain

What is the relationship between sentence representations learned by dee...
research
07/06/2023

UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language

Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our un...
research
09/18/2020

Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models

We report a GPT-based multi-sentence language model for dialogue generat...
research
03/27/2023

Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain

Linking computational natural language processing (NLP) models and neura...
research
09/19/2022

On the benefits of self-taught learning for brain decoding

We study the benefits of using a large public neuroimaging database comp...

Please sign up or login with your details

Forgot password? Click here to reset