Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction

10/05/2019
by   Maosen Li, et al.
0

3D skeleton-based action recognition and motion prediction are two essential problems of human activity understanding. In many previous works: 1) they studied two tasks separately, neglecting internal correlations; 2) they did not capture sufficient relations inside the body. To address these issues, we propose a symbiotic model to handle two tasks jointly; and we propose two scales of graphs to explicitly capture relations among body-joints and body-parts. Together, we propose symbiotic graph neural networks, which contain a backbone, an action-recognition head, and a motion-prediction head. Two heads are trained jointly and enhance each other. For the backbone, we propose multi-branch multi-scale graph convolution networks to extract spatial and temporal features. The multi-scale graph convolution networks are based on joint-scale and part-scale graphs. The joint-scale graphs contain actional graphs, capturing action-based relations, and structural graphs, capturing physical constraints. The part-scale graphs integrate body-joints to form specific parts, representing high-level relations. Moreover, dual bone-based graphs and networks are proposed to learn complementary features. We conduct extensive experiments for skeleton-based action recognition and motion prediction with four datasets, NTU-RGB+D, Kinetics, Human3.6M, and CMU Mocap. Experiments show that our symbiotic graph neural networks achieve better performances on both tasks compared to the state-of-the-art methods.

READ FULL TEXT
research
04/26/2019

Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition

Action recognition with skeleton data has recently attracted much attent...
research
12/05/2020

Multi Scale Temporal Graph Networks For Skeleton-based Action Recognition

Graph convolutional networks (GCNs) can effectively capture the features...
research
07/05/2021

SM-SGE: A Self-Supervised Multi-Scale Skeleton Graph Encoding Framework for Person Re-Identification

Person re-identification via 3D skeletons is an emerging topic with grea...
research
11/17/2017

Action-Attending Graphic Neural Network

The motion analysis of human skeletons is crucial for human action recog...
research
03/17/2020

Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction

We propose novel dynamic multiscale graph neural networks (DMGNN) to pre...
research
07/31/2022

Skeleton-Parted Graph Scattering Networks for 3D Human Motion Prediction

Graph convolutional network based methods that model the body-joints' re...
research
03/14/2023

Simultaneous Action Recognition and Human Whole-Body Motion and Dynamics Prediction from Wearable Sensors

This paper presents a novel approach to solve simultaneously the problem...

Please sign up or login with your details

Forgot password? Click here to reset