DeepAI AI Chat
Log In Sign Up

A visual encoding model based on deep neural networks and transfer learning

by   Chi Zhang, et al.

Background: Building visual encoding models to accurately predict visual responses is a central challenge for current vision-based brain-machine interface techniques. To achieve high prediction accuracy on neural signals, visual encoding models should include precise visual features and appropriate prediction algorithms. Most existing visual encoding models employ hand-craft visual features (e.g., Gabor wavelets or semantic labels) or data-driven features (e.g., features extracted from deep neural networks (DNN)). They also assume a linear mapping between feature representation to brain activity. However, it remains unknown whether such linear mapping is sufficient for maximizing prediction accuracy. New Method: We construct a new visual encoding framework to predict cortical responses in a benchmark functional magnetic resonance imaging (fMRI) dataset. In this framework, we employ the transfer learning technique to incorporate a pre-trained DNN (i.e., AlexNet) and train a nonlinear mapping from visual features to brain activity. This nonlinear mapping replaces the conventional linear mapping and is supposed to improve prediction accuracy on brain activity. Results: The proposed framework can significantly predict responses of over 20 V1-lateral occipital region, LO) and achieve unprecedented prediction accuracy. Comparison with Existing Methods: Comparing to two conventional visual encoding models, we find that the proposed encoding model shows consistent higher prediction accuracy in all early visual areas, especially in relatively anterior visual areas (i.e., V4 and LO). Conclusions: Our work proposes a new framework to utilize pre-trained visual features and train non-linear mappings from visual features to brain activity.


page 4

page 8

page 14


Decoding Neural Responses in Mouse Visual Cortex through a Deep Neural Network

Finding a code to unravel the population of neural responses that leads ...

Comparing Neural and Attractiveness-based Visual Features for Artwork Recommendation

Advances in image processing and computer vision in the latest years hav...

Neural encoding and interpretation for high-level visual cortices based on fMRI using image caption features

On basis of functional magnetic resonance imaging (fMRI), researchers ar...

Visual Encoding and Debiasing for CTR Prediction

Extracting expressive visual features is crucial for accurate Click-Thro...

Effective and efficient ROI-wise visual encoding using an end-to-end CNN regression model and selective optimization

Recently, visual encoding based on functional magnetic resonance imaging...

Relating CNNs with brain: Challenges and findings

Conventional neural network models (CNN), loosely inspired by the primat...