Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction

08/25/2021
by   Maosen Li, et al.
0

We propose a multiscale spatio-temporal graph neural network (MST-GNN) to predict the future 3D skeleton-based human poses in an action-category-agnostic manner. The core of MST-GNN is a multiscale spatio-temporal graph that explicitly models the relations in motions at various spatial and temporal scales. Different from many previous hierarchical structures, our multiscale spatio-temporal graph is built in a data-adaptive fashion, which captures nonphysical, yet motion-based relations. The key module of MST-GNN is a multiscale spatio-temporal graph computational unit (MST-GCU) based on the trainable graph structure. MST-GCU embeds underlying features at individual scales and then fuses features across scales to obtain a comprehensive representation. The overall architecture of MST-GNN follows an encoder-decoder framework, where the encoder consists of a sequence of MST-GCUs to learn the spatial and temporal features of motions, and the decoder uses a graph-based attention gate recurrent unit (GA-GRU) to generate future poses. Extensive experiments are conducted to show that the proposed MST-GNN outperforms state-of-the-art methods in both short and long-term motion prediction on the datasets of Human 3.6M, CMU Mocap and 3DPW, where MST-GNN outperforms previous works by 5.33 long-term prediction on Human 3.6M, and by 11.84 errors for short-term and long-term prediction on CMU Mocap, and by 1.13 mean angle errors on 3DPW in average, respectively. We further investigate the learned multiscale graphs for interpretability.

READ FULL TEXT

page 1

page 15

research
03/17/2020

Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction

We propose novel dynamic multiscale graph neural networks (DMGNN) to pre...
research
11/06/2019

Predicting Long-Term Skeletal Motions by a Spatio-Temporal Hierarchical Recurrent Network

The primary goal of skeletal motion prediction is to generate future mot...
research
09/21/2021

Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs

Several applications such as autonomous driving, augmented reality and v...
research
01/22/2022

Predicting Physics in Mesh-reduced Space with Temporal Attention

Graph-based next-step prediction models have recently been very successf...
research
04/10/2020

Multiresolution Convolutional Autoencoders

We propose a multi-resolution convolutional autoencoder (MrCAE) architec...
research
12/20/2021

DMS-GCN: Dynamic Mutiscale Spatiotemporal Graph Convolutional Networks for Human Motion Prediction

Human motion prediction is an important and challenging task in many com...
research
08/02/2023

Spatio-Temporal Branching for Motion Prediction using Motion Increments

Human motion prediction (HMP) has emerged as a popular research topic du...

Please sign up or login with your details

Forgot password? Click here to reset