Learning to Forecast Videos of Human Activity with Multi-granularity Models and Adaptive Rendering

12/05/2017
by   Mengyao Zhai, et al.
0

We propose an approach for forecasting video of complex human activity involving multiple people. Direct pixel-level prediction is too simple to handle the appearance variability in complex activities. Hence, we develop novel intermediate representations. An architecture combining a hierarchical temporal model for predicting human poses and encoder-decoder convolutional neural networks for rendering target appearances is proposed. Our hierarchical model captures interactions among people by adopting a dynamic group-based interaction mechanism. Next, our appearance rendering network encodes the targets' appearances by learning adaptive appearance filters using a fully convolutional network. Finally, these filters are placed in encoder-decoder neural networks to complete the rendering. We demonstrate that our model can generate videos that are superior to state-of-the-art methods, and can handle complex human activity scenarios in video forecasting.

READ FULL TEXT

page 1

page 3

page 4

page 7

page 8

research
04/24/2021

Adaptive Appearance Rendering

We propose an approach to generate images of people given a desired appe...
research
06/25/2017

Decomposing Motion and Content for Natural Video Sequence Prediction

We propose a deep neural network for the prediction of future frames in ...
research
12/10/2019

Forecasting Future Sequence of Actions to Complete an Activity

Future human action forecasting from partial observations of activities ...
research
11/29/2022

Encoder-Decoder Model for Suffix Prediction in Predictive Monitoring

Predictive monitoring is a subfield of process mining that aims to predi...
research
03/03/2017

Learning Robot Activities from First-Person Human Videos Using Convolutional Future Regression

We design a new approach that allows robot learning of new activities fr...
research
01/26/2015

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

Human activity understanding with 3D/depth sensors has received increasi...
research
08/13/2018

Time Perception Machine: Temporal Point Processes for the When, Where and What of Activity Prediction

Numerous powerful point process models have been developed to understand...

Please sign up or login with your details

Forgot password? Click here to reset