Accurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture

01/02/2018
by   Kai Qiao, et al.
0

In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of the deep network computation. However, comparing with the goal of decoding orientation, position and object category from activities in visual cortex, accurate reconstruction of image stimuli from human fMRI is a still challenging work. In this paper, the capsule network (CapsNet) architecture based visual reconstruction (CNAVR) method is developed to reconstruct image stimuli. The capsule means containing a group of neurons to perform the better organization of feature structure and representation, inspired by the structure of cortical mini column including several hundred neurons in primates. The high-level capsule features in the CapsNet includes diverse features of image stimuli such as semantic class, orientation, location and so on. We used these features to bridge between human fMRI and image stimuli. We firstly employed the CapsNet to train the nonlinear mapping from image stimuli to high-level capsule features, and from high-level capsule features to image stimuli again in an end-to-end manner. After estimating the serviceability of each voxel by encoding performance to accomplish the selecting of voxels, we secondly trained the nonlinear mapping from dimension-decreasing fMRI data to high-level capsule features. Finally, we can predict the high-level capsule features with fMRI data, and reconstruct image stimuli with the CapsNet. We evaluated the proposed CNAVR method on the dataset of handwritten digital images, and exceeded about 10 existing state-of-the-art methods on the structural similarity index (SSIM).

READ FULL TEXT

page 8

page 9

page 10

research
03/26/2020

Neural encoding and interpretation for high-level visual cortices based on fMRI using image caption features

On basis of functional magnetic resonance imaging (fMRI), researchers ar...
research
01/16/2018

Constraint-free Natural Image Reconstruction from fMRI Signals Based on Convolutional Neural Network

In recent years, research on decoding brain activity based on functional...
research
04/23/2019

Corticospinal Tract (CST) reconstruction based on fiber orientation distributions(FODs) tractography

The Corticospinal Tract (CST) is a part of pyramidal tract (PT), and it ...
research
07/27/2019

Effective and efficient ROI-wise visual encoding using an end-to-end CNN regression model and selective optimization

Recently, visual encoding based on functional magnetic resonance imaging...
research
05/25/2020

Optimizing Visual Cortex Parameterization with Error-Tolerant Teichmuller Map in Retinotopic Mapping

The mapping between the visual input on the retina to the cortical surfa...
research
12/26/2016

Multi-Region Neural Representation: A novel model for decoding visual stimuli in human brains

Multivariate Pattern (MVP) classification holds enormous potential for d...
research
09/04/2016

Decoding visual stimuli in human brain by using Anatomical Pattern Analysis on fMRI images

A universal unanswered question in neuroscience and machine learning is ...

Please sign up or login with your details

Forgot password? Click here to reset