Deep Learning Human Mind for Automated Visual Classification

09/01/2016
by   Concetto Spampinato, et al.
0

What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories. Afterwards, we train a Convolutional Neural Network (CNN)-based regressor to project images onto the learned manifold, thus effectively allowing machines to employ human brain-based features for automated visual classification. We use a 32-channel EEG to record brain activity of seven subjects while looking at images of 40 ImageNet object classes. The proposed RNN based approach for discriminating object classes using brain signals reaches an average accuracy of about 40 which outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain-driven approach obtains competitive performance, comparable to those achieved by powerful CNN models, both on ImageNet and CalTech 101, thus demonstrating its classification and generalization capabilities. This gives us a real hope that, indeed, human mind can be read and transferred to machines.

READ FULL TEXT
research
11/17/2022

Brain informed transfer learning for categorizing construction hazards

A transfer learning paradigm is proposed for "knowledge" transfer betwee...
research
11/01/2018

Cogni-Net: Cognitive Feature Learning through Deep Visual Perception

Can we ask computers to recognize what we see from brain signals alone? ...
research
11/26/2018

Using Variable Natural Environment Brain-Computer Interface Stimuli for Real-time Humanoid Robot Navigation

This paper addresses the challenge of humanoid robot teleoperation in a ...
research
01/11/2022

Where Is My Mind (looking at)? Predicting Visual Attention from Brain Activity

Visual attention estimation is an active field of research at the crossr...
research
10/25/2018

Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features

This paper tackles the problem of learning brain-visual representations ...
research
12/18/2018

Training on the test set? An analysis of Spampinato et al. [31]

A recent paper [31] claims to classify brain processing evoked in subjec...
research
09/11/2018

Taking a machine's perspective: Human deciphering of adversarial images

How similar is the human mind to the sophisticated machine-learning syst...

Please sign up or login with your details

Forgot password? Click here to reset