Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

11/01/2021
by   Jianing Qiu, et al.
0

In this paper, we address the problem of forecasting the trajectory of an egocentric camera wearer (ego-person) in crowded spaces. The trajectory forecasting ability learned from the data of different camera wearers walking around in the real world can be transferred to assist visually impaired people in navigation, as well as to instill human navigation behaviours in mobile robots, enabling better human-robot interactions. To this end, a novel egocentric human trajectory forecasting dataset was constructed, containing real trajectories of people navigating in crowded spaces wearing a camera, as well as extracted rich contextual data. We extract and utilize three different modalities to forecast the trajectory of the camera wearer, i.e., his/her past trajectory, the past trajectories of nearby people, and the environment such as the scene semantics or the depth of the scene. A Transformer-based encoder-decoder neural network model, integrated with a novel cascaded cross-attention mechanism that fuses multiple modalities, has been designed to predict the future trajectory of the camera wearer. Extensive experiments have been conducted, and the results have shown that our model outperforms the state-of-the-art methods in egocentric human trajectory forecasting.

READ FULL TEXT

page 1

page 2

research
03/06/2021

Indoor Future Person Localization from an Egocentric Wearable Camera

Accurate prediction of future person location and movement trajectory fr...
research
10/17/2022

Forecasting Human Trajectory from Scene History

Predicting the future trajectory of a person remains a challenging probl...
research
03/22/2022

Under the Hood of Transformer Networks for Trajectory Forecasting

Transformer Networks have established themselves as the de-facto state-o...
research
11/10/2020

Simple means Faster: Real-Time Human Motion Forecasting in Monocular First Person Videos on CPU

We present a simple, fast, and light-weight RNN based framework for fore...
research
08/10/2021

Multi-Camera Trajectory Forecasting with Trajectory Tensors

We introduce the problem of multi-camera trajectory forecasting (MCTF), ...
research
04/08/2021

Panoptic Segmentation Forecasting

Our goal is to forecast the near future given a set of recent observatio...
research
12/07/2018

Back to square one: probabilistic trajectory forecasting without bells and whistles

We introduce a spatio-temporal convolutional neural network model for tr...

Please sign up or login with your details

Forgot password? Click here to reset