Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN

by   Milad Mozafari, et al.

Decoding and reconstructing images from brain imaging data is a research area of high interest. Recent progress in deep generative neural networks has introduced new opportunities to tackle this problem. Here, we employ a recently proposed large-scale bi-directional generative adversarial network, called BigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. BigBiGAN converts images into a 120-dimensional latent space which encodes class and attribute information together, and can also reconstruct images based on their latent vectors. We trained a linear mapping between fMRI data, acquired over images from 150 different categories of ImageNet, and their corresponding BigBiGAN latent vectors. Then, we applied this mapping to the fMRI activity patterns obtained from 50 new test images from 50 unseen categories in order to retrieve their latent vectors, and reconstruct the corresponding images. Pairwise image decoding from the predicted latent vectors was highly accurate (84 resulting image reconstructions were visually plausible, successfully captured many attributes of the original images, and had high perceptual similarity with the original content. This method establishes a new state-of-the-art for fMRI-based natural image reconstruction, and can be flexibly updated to take into account any future improvements in generative models of natural scene images.


page 1

page 2

page 4

page 6


Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks

While objects from different categories can be reliably decoded from fMR...

Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors

We present MindEye, a novel fMRI-to-image approach to retrieve and recon...

Semantic Neural Decoding via Cross-Modal Generation

Semantic neural decoding aims to elucidate the cognitive processes of th...

Reconstruction of Perceived Images from fMRI Patterns and Semantic Brain Exploration using Instance-Conditioned GANs

Reconstructing perceived natural images from fMRI signals is one of the ...

Sound reconstruction from human brain activity via a generative model with brain-like auditory features

The successful reconstruction of perceptual experiences from human brain...

Reconstruction of Natural Visual Scenes from Neural Spikes with Deep Neural Networks

Neural coding is one of the central questions in systems neuroscience fo...

From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Reconstructing observed images from fMRI brain recordings is challenging...

Please sign up or login with your details

Forgot password? Click here to reset