Neural Photofit: Gaze-based Mental Image Reconstruction

08/17/2021
by   Florian Strohm, et al.
0

We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer's mental image.

READ FULL TEXT

page 3

page 6

page 8

research
11/27/2016

It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation

Eye gaze is an important non-verbal cue for human affect analysis. Recen...
research
06/19/2017

Visual Decoding of Targets During Visual Search From Human Eye Fixations

What does human gaze reveal about a users' intents and to which extend c...
research
10/09/2020

Controllable Continuous Gaze Redirection

In this work, we present interpGaze, a novel framework for controllable ...
research
05/25/2023

Introducing Explicit Gaze Constraints to Face Swapping

Face swapping combines one face's identity with another face's non-appea...
research
11/01/2020

FusiformNet: Extracting Discriminative Facial Features on Different Levels

Over the last several years, research on facial recognition based on Dee...
research
12/19/2021

A New Image Codec Paradigm for Human and Machine Uses

With the AI of Things (AIoT) development, a huge amount of visual data, ...

Please sign up or login with your details

Forgot password? Click here to reset