More than meets the eye: Self-supervised depth reconstruction from brain activity

06/09/2021
by   Guy Gaziv, et al.
0

In the past few years, significant advancements were made in reconstruction of observed natural images from fMRI brain recordings using deep-learning tools. Here, for the first time, we show that dense 3D depth maps of observed 2D natural images can also be recovered directly from fMRI brain recordings. We use an off-the-shelf method to estimate the unknown depth maps of natural images. This is applied to both: (i) the small number of images presented to subjects in an fMRI scanner (images for which we have fMRI recordings - referred to as "paired" data), and (ii) a very large number of natural images with no fMRI recordings ("unpaired data"). The estimated depth maps are then used as an auxiliary reconstruction criterion to train for depth reconstruction directly from fMRI. We propose two main approaches: Depth-only recovery and joint image-depth RGBD recovery. Because the number of available "paired" training data (images with fMRI) is small, we enrich the training data via self-supervised cycle-consistent training on many "unpaired" data (natural images depth maps without fMRI). This is achieved using our newly defined and trained Depth-based Perceptual Similarity metric as a reconstruction criterion. We show that predicting the depth map directly from fMRI outperforms its indirect sequential recovery from the reconstructed images. We further show that activations from early cortical visual areas dominate our depth reconstruction results, and propose means to characterize fMRI voxels by their degree of depth-information tuning. This work adds an important layer of decoded information, extending the current envelope of visual brain decoding capabilities.

READ FULL TEXT

page 2

page 3

page 4

page 8

research
06/07/2022

A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity

Reconstructing natural videos from fMRI brain recordings is very challen...
research
07/03/2019

From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Reconstructing observed images from fMRI brain recordings is challenging...
research
09/12/2016

Examining Representational Similarity in ConvNets and the Primate Visual Cortex

We compare several ConvNets with different depth and regularization tech...
research
02/25/2023

BrainCLIP: Bridging Brain and Visual-Linguistic Representation via CLIP for Generic Natural Visual Stimulus Decoding from fMRI

Reconstructing perceived natural images or decoding their categories fro...
research
11/16/2019

Transfer Learning of fMRI Dynamics

As a mental disorder progresses, it may affect brain structure, but brai...
research
10/23/2012

On the geometric structure of fMRI searchlight-based information maps

Information mapping is a popular application of Multivoxel Pattern Analy...
research
06/16/2023

DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding

The human brain possesses remarkable abilities in visual processing, inc...

Please sign up or login with your details

Forgot password? Click here to reset