DeepAI AI Chat
Log In Sign Up

Neural encoding and interpretation for high-level visual cortices based on fMRI using image caption features

by   Kai Qiao, et al.

On basis of functional magnetic resonance imaging (fMRI), researchers are devoted to designing visual encoding models to predict the neuron activity of human in response to presented image stimuli and analyze inner mechanism of human visual cortices. Deep network structure composed of hierarchical processing layers forms deep network models by learning features of data on specific task through big dataset. Deep network models have powerful and hierarchical representation of data, and have brought about breakthroughs for visual encoding, while revealing hierarchical structural similarity with the manner of information processing in human visual cortices. However, previous studies almost used image features of those deep network models pre-trained on classification task to construct visual encoding models. Except for deep network structure, the task or corresponding big dataset is also important for deep network models, but neglected by previous studies. Because image classification is a relatively fundamental task, it is difficult to guide deep network models to master high-level semantic representations of data, which causes into that encoding performance for high-level visual cortices is limited. In this study, we introduced one higher-level vision task: image caption (IC) task and proposed the visual encoding model based on IC features (ICFVEM) to encode voxels of high-level visual cortices. Experiment demonstrated that ICFVEM obtained better encoding performance than previous deep network models pre-trained on classification task. In addition, the interpretation of voxels was realized to explore the detailed characteristics of voxels based on the visualization of semantic words, and comparative analysis implied that high-level visual cortices behaved the correlative representation of image content.


page 6

page 12

page 13

page 14

page 15


Effective and efficient ROI-wise visual encoding using an end-to-end CNN regression model and selective optimization

Recently, visual encoding based on functional magnetic resonance imaging...

Accurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture

In neuroscience, all kinds of computation models were designed to answer...

A visual encoding model based on deep neural networks and transfer learning

Background: Building visual encoding models to accurately predict visual...

Neural encoding with visual attention

Visual perception is critically influenced by the focus of attention. Du...

Categorical Difference and Related Brain Regions of the Attentional Blink Effect

Attentional blink (AB) is a biological effect, showing that for 200 to 5...

Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds

Recent studies have shown that aggregating convolutional features of a p...