Semi-supervised Body Parsing and Pose Estimation for Enhancing Infant General Movement Assessment

by   Haomiao Ni, et al.

General movement assessment (GMA) of infant movement videos (IMVs) is an effective method for early detection of cerebral palsy (CP) in infants. We demonstrate in this paper that end-to-end trainable neural networks for image sequence recognition can be applied to achieve good results in GMA, and more importantly, augmenting raw video with infant body parsing and pose estimation information can significantly improve performance. To solve the problem of efficiently utilizing partially labeled IMVs for body parsing, we propose a semi-supervised model, termed SiamParseNet (SPN), which consists of two branches, one for intra-frame body parts segmentation and another for inter-frame label propagation. During training, the two branches are jointly trained by alternating between using input pairs of only labeled frames and input of both labeled and unlabeled frames. We also investigate training data augmentation by proposing a factorized video generative adversarial network (FVGAN) to synthesize novel labeled frames for training. When testing, we employ a multi-source inference mechanism, where the final result for a test frame is either obtained via the segmentation branch or via propagation from a nearby key frame. We conduct extensive experiments for body parsing using SPN on two infant movement video datasets, where SPN coupled with FVGAN achieves state-of-the-art performance. We further demonstrate that SPN can be easily adapted to the infant pose estimation task with superior performance. Last but not least, we explore the clinical application of our method for GMA. We collected a new clinical IMV dataset with GMA annotations, and our experiments show that SPN models for body parsing and pose estimation trained on the first two datasets generalize well to the new clinical dataset and their results can significantly boost the CRNN-based GMA prediction performance.


page 1

page 2

page 3

page 5

page 7

page 9

page 11

page 13


SiamParseNet: Joint Body Parsing and Label Propagation in Infant Movement Videos

General movement assessment (GMA) of infant movement videos (IMVs) is an...

SemiMultiPose: A Semi-supervised Multi-animal Pose Estimation Framework

Multi-animal pose estimation is essential for studying animals' social b...

Learning Temporal Pose Estimation from Sparsely-Labeled Videos

Modern approaches for multi-person pose estimation in video require larg...

Semi-Supervised 3D Hand Shape and Pose Estimation with Label Propagation

To obtain 3D annotations, we are restricted to controlled environments o...

Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer

Human body part parsing, or human semantic part segmentation, is fundame...

Pose-MUM : Reinforcing Key Points Relationship for Semi-Supervised Human Pose Estimation

A well-designed strong-weak augmentation strategy and the stable teacher...

Slim DensePose: Thrifty Learning from Sparse Annotations and Motion Cues

DensePose supersedes traditional landmark detectors by densely mapping i...

Please sign up or login with your details

Forgot password? Click here to reset