Towards an Embodied Semantic Fovea: Semantic 3D scene reconstruction from ego-centric eye-tracker videos

07/27/2018
by   Mickey Li, et al.
0

Incorporating the physical environment is essential for a complete understanding of human behavior in unconstrained every-day tasks. This is especially important in ego-centric tasks where obtaining 3 dimensional information is both limiting and challenging with the current 2D video analysis methods proving insufficient. Here we demonstrate a proof-of-concept system which provides real-time 3D mapping and semantic labeling of the local environment from an ego-centric RGB-D video-stream with 3D gaze point estimation from head mounted eye tracking glasses. We augment existing work in Semantic Simultaneous Localization And Mapping (Semantic SLAM) with collected gaze vectors. Our system can then find and track objects both inside and outside the user field-of-view in 3D from multiple perspectives with reasonable accuracy. We validate our concept by producing a semantic map from images of the NYUv2 dataset while simultaneously estimating gaze position and gaze classes from recorded gaze data of the dataset images.

READ FULL TEXT

page 2

page 3

page 4

research
04/21/2022

Single-User 2D Gaze Estimation in Retail Environment Using Deep Learning

Human gaze estimation is a widely used technique to observe human behavi...
research
05/27/2020

Eye Gaze Controlled Interfaces for Head Mounted and Multi-Functional Displays in Military Aviation Environment

Eye gaze controlled interfaces allow us to directly manipulate a graphic...
research
01/11/2016

3D Gaze Estimation from 2D Pupil Positions on Monocular Head-Mounted Eye Trackers

3D gaze information is important for scene-centric attention analysis bu...
research
07/30/2013

An Integrated System for 3D Gaze Recovery and Semantic Analysis of Human Attention

This work describes a computer vision system that enables pervasive mapp...
research
08/24/2019

EyeNet: A Multi-Task Network for Off-Axis Eye Gaze Estimation and User Understanding

Eye gaze estimation and simultaneous semantic understanding of a user th...
research
08/08/2022

Where Are You Looking?: A Large-Scale Dataset of Head and Gaze Behavior for 360-Degree Videos and a Pilot Study

360 videos in recent years have experienced booming development. Compare...
research
04/15/2019

Custom Video-Oculography Device and Its Application to Fourth Purkinje Image Detection during Saccades

We built a custom video-based eye-tracker that saves every video frame a...

Please sign up or login with your details

Forgot password? Click here to reset