Mapping Between fMRI Responses to Movies and their Natural Language Annotations

10/13/2016
by   Kiran Vodrahalli, et al.
0

Several research groups have shown how to correlate fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock [1], and learn bidirectional mappings between fMRI responses and natural language representations. We show how to leverage data from multiple subjects watching the same movie to improve the accuracy of the mappings, allowing us to succeed at a scene classification task with 72 guessing would give 4 top 4 the Shared Response Model (SRM) and its variant SRM-ICA [2, 3] to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature [4] that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data.

READ FULL TEXT

page 11

page 13

page 14

research
06/09/2021

Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses

How related are the representations learned by neural language models, t...
research
09/10/2020

Brain2Word: Decoding Brain Activity for Language Generation

Brain decoding, understood as the process of mapping brain activities to...
research
09/01/2019

Incidental Supervision from Question-Answering Signals

Human annotations are costly for many natural language processing (NLP) ...
research
05/19/2023

Scaling laws for language encoding models in fMRI

Representations from transformer-based unidirectional language models ar...
research
09/27/2019

Fast shared response model for fMRI data

The shared response model provides a simple but effective framework toan...
research
03/05/2015

Pyrcca: regularized kernel canonical correlation analysis in Python and its applications to neuroimaging

Canonical correlation analysis (CCA) is a valuable method for interpreti...
research
07/27/2020

Characterizing the Effect of Sentence Context on Word Meanings: Mapping Brain to Behavior

Semantic feature models have become a popular tool for prediction and in...

Please sign up or login with your details

Forgot password? Click here to reset