Ego-Pose Estimation and Forecasting as Real-Time PD Control

06/07/2019
by   Ye Yuan, et al.
13

We propose the use of a proportional-derivative (PD) control based policy learned via reinforcement learning (RL) to estimate and forecast 3D human pose from egocentric videos. The method learns directly from unsegmented egocentric videos and motion capture data consisting of various complex human motions (e.g., crouching, hopping, bending, and motion transitions). We propose a video-conditioned recurrent control technique to forecast physically-valid and stable future motions of arbitrary length. We also introduce a value function based fail-safe mechanism which enables our method to run as a single pass algorithm over the video data. Experiments with both controlled and in-the-wild data show that our approach outperforms previous art in both quantitative metrics and visual quality of the motions, and is also robust enough to transfer directly to real-world scenarios. Additionally, our time analysis shows that the combined use of our pose estimation and forecasting can run at 30 FPS, making it suitable for real-time applications.

READ FULL TEXT

page 1

page 4

page 5

page 7

research
03/19/2020

3D Ego-Pose Estimation via Imitation Learning

Ego-pose estimation, i.e., estimating a person's 3D pose with a single w...
research
08/02/2022

UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture

We present UnrealEgo, i.e., a new large-scale naturalistic dataset for e...
research
07/09/2022

Snipper: A Spatiotemporal Transformer for Simultaneous Multi-Person 3D Pose Estimation Tracking and Forecasting on a Video Snippet

Multi-person pose understanding from RGB videos includes three complex t...
research
07/26/2018

Learning to Forecast and Refine Residual Motion for Image-to-Video Generation

We consider the problem of image-to-video translation, where an input im...
research
08/02/2015

Recurrent Network Models for Human Dynamics

We propose the Encoder-Recurrent-Decoder (ERD) model for recognition and...
research
02/12/2019

Post-Data Augmentation to Improve Deep Pose Estimation of Extreme and Wild Motions

Contributions of recent deep-neural-network (DNN) based techniques have ...
research
08/24/2023

Robot Pose Nowcasting: Forecast the Future to Improve the Present

In recent years, the effective and safe collaboration between humans and...

Please sign up or login with your details

Forgot password? Click here to reset