HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions

03/17/2022
by   Han Liang, et al.
5

Monocular 3D motion capture (mocap) is beneficial to many applications. The use of a single camera, however, often fails to handle occlusions of different body parts and hence it is limited to capture relatively simple movements. We present a light-weight, hybrid mocap technique called HybridCap that augments the camera with only 4 Inertial Measurement Units (IMUs) in a learning-and-optimization framework. We first employ a weakly-supervised and hierarchical motion inference module based on cooperative Gated Recurrent Unit (GRU) blocks that serve as limb, body and root trackers as well as an inverse kinematics solver. Our network effectively narrows the search space of plausible motions via coarse-to-fine pose estimation and manages to tackle challenging movements with high efficiency. We further develop a hybrid optimization scheme that combines inertial feedback and visual cues to improve tracking accuracy. Extensive experiments on various datasets demonstrate HybridCap can robustly handle challenging movements ranging from fitness actions to Latin dance. It also achieves real-time performance up to 60 fps with state-of-the-art accuracy.

READ FULL TEXT

page 5

page 9

page 10

page 12

page 13

research
03/16/2022

Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors

Motion capture from sparse inertial sensors has shown great potential co...
research
05/18/2020

Designing Just-in-Time Detection for Gamified Fitness Frameworks

This paper presents our findings from a multi-year effort to detect moti...
research
02/21/2020

Particle Filter Based Monocular Human Tracking with a 3D Cardbox Model and a Novel Deterministic Resampling Strategy

The challenge of markerless human motion tracking is the high dimensiona...
research
09/01/2023

Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture

Either RGB images or inertial signals have been used for the task of mot...
research
03/11/2021

ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References

Capturing challenging human motions is critical for numerous application...
research
03/04/2020

MoVi: A Large Multipurpose Motion and Video Dataset

Human movements are both an area of intense study and the basis of many ...
research
07/03/2023

Real-time Monocular Full-body Capture in World Space via Sequential Proxy-to-Motion Learning

Learning-based approaches to monocular motion capture have recently show...

Please sign up or login with your details

Forgot password? Click here to reset