DeepAI AI Chat
Log In Sign Up

Accurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture

by   Kai Qiao, et al.

In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of the deep network computation. However, comparing with the goal of decoding orientation, position and object category from activities in visual cortex, accurate reconstruction of image stimuli from human fMRI is a still challenging work. In this paper, the capsule network (CapsNet) architecture based visual reconstruction (CNAVR) method is developed to reconstruct image stimuli. The capsule means containing a group of neurons to perform the better organization of feature structure and representation, inspired by the structure of cortical mini column including several hundred neurons in primates. The high-level capsule features in the CapsNet includes diverse features of image stimuli such as semantic class, orientation, location and so on. We used these features to bridge between human fMRI and image stimuli. We firstly employed the CapsNet to train the nonlinear mapping from image stimuli to high-level capsule features, and from high-level capsule features to image stimuli again in an end-to-end manner. After estimating the serviceability of each voxel by encoding performance to accomplish the selecting of voxels, we secondly trained the nonlinear mapping from dimension-decreasing fMRI data to high-level capsule features. Finally, we can predict the high-level capsule features with fMRI data, and reconstruct image stimuli with the CapsNet. We evaluated the proposed CNAVR method on the dataset of handwritten digital images, and exceeded about 10 existing state-of-the-art methods on the structural similarity index (SSIM).


page 8

page 9

page 10


Neural encoding and interpretation for high-level visual cortices based on fMRI using image caption features

On basis of functional magnetic resonance imaging (fMRI), researchers ar...

Corticospinal Tract (CST) reconstruction based on fiber orientation distributions(FODs) tractography

The Corticospinal Tract (CST) is a part of pyramidal tract (PT), and it ...

Effective and efficient ROI-wise visual encoding using an end-to-end CNN regression model and selective optimization

Recently, visual encoding based on functional magnetic resonance imaging...

Optimizing Visual Cortex Parameterization with Error-Tolerant Teichmuller Map in Retinotopic Mapping

The mapping between the visual input on the retina to the cortical surfa...

Multi-Region Neural Representation: A novel model for decoding visual stimuli in human brains

Multivariate Pattern (MVP) classification holds enormous potential for d...

Mind Reader: Reconstructing complex images from brain activities

Understanding how the brain encodes external stimuli and how these stimu...

Decoding natural image stimuli from fMRI data with a surface-based convolutional network

Due to the low signal-to-noise ratio and limited resolution of functiona...