Facial Image Reconstruction from Functional Magnetic Resonance Imaging via GAN Inversion with Improved Attribute Consistency

07/03/2022
by   Pei-Chun Chang, et al.
0

Neuroscience studies have revealed that the brain encodes visual content and embeds information in neural activity. Recently, deep learning techniques have facilitated attempts to address visual reconstructions by mapping brain activity to image stimuli using generative adversarial networks (GANs). However, none of these studies have considered the semantic meaning of latent code in image space. Omitting semantic information could potentially limit the performance. In this study, we propose a new framework to reconstruct facial images from functional Magnetic Resonance Imaging (fMRI) data. With this framework, the GAN inversion is first applied to train an image encoder to extract latent codes in image space, which are then bridged to fMRI data using linear transformation. Following the attributes identified from fMRI data using an attribute classifier, the direction in which to manipulate attributes is decided and the attribute manipulator adjusts the latent code to improve the consistency between the seen image and the reconstructed image. Our experimental results suggest that the proposed framework accomplishes two goals: (1) reconstructing clear facial images from fMRI data and (2) maintaining the consistency of semantic characteristics.

READ FULL TEXT

page 1

page 5

page 7

research
03/13/2020

BigGAN-based Bayesian reconstruction of natural images from human brain activity

In the visual decoding domain, visually reconstructing presented images ...
research
05/25/2020

Optimizing Visual Cortex Parameterization with Error-Tolerant Teichmuller Map in Retinotopic Mapping

The mapping between the visual input on the retina to the cortical surfa...
research
09/30/2022

Mind Reader: Reconstructing complex images from brain activities

Understanding how the brain encodes external stimuli and how these stimu...
research
05/19/2017

Deep adversarial neural decoding

Here, we present a novel approach to solve the problem of reconstructing...
research
05/17/2023

Controllable Mind Visual Diffusion Model

Brain signal visualization has emerged as an active research area, servi...
research
01/13/2019

DCNN-GAN: Reconstructing Realistic Image from fMRI

Visualizing the perceptual content by analyzing human functional magneti...
research
01/28/2021

Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

Reconstructing seeing images from fMRI recordings is an absorbing resear...

Please sign up or login with your details

Forgot password? Click here to reset