Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction

09/11/2023

∙

We present Blendshapes GHUM, an on-device ML pipeline that predicts 52 facial blendshape coefficients at 30+ FPS on modern mobile phones, from a single monocular RGB image and enables facial motion capture applications like virtual avatars. Our main contributions are: i) an annotation-free offline method for obtaining blendshape coefficients from real-world human scans, ii) a lightweight real-time model that predicts blendshape coefficients based on facial landmarks.

READ FULL TEXT

Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction

Real-time Pupil Tracking from Monocular Video for Digital Puppetry

BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation

4D Agnostic Real-Time Facial Animation Pipeline for Desktop Scenarios

Real-time 3D Facial Tracking via Cascaded Compositional Learning

Real-Time Cleaning and Refinement of Facial Animation Signals

Attention Mesh: High-fidelity Face Mesh Prediction in Real-time

Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction

Related Research

Real-time Pupil Tracking from Monocular Video for Digital Puppetry

BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation

4D Agnostic Real-Time Facial Animation Pipeline for Desktop Scenarios

Real-time 3D Facial Tracking via Cascaded Compositional Learning

Real-Time Cleaning and Refinement of Facial Animation Signals

Attention Mesh: High-fidelity Face Mesh Prediction in Real-time

Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs