Learning Individual Styles of Conversational Gesture

06/10/2019
by   Shiry Ginosar, et al.
11

Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild" monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at http://people.eecs.berkeley.edu/ shiry/speech2gesture .

READ FULL TEXT

page 1

page 2

page 6

page 8

page 11

research
08/05/2022

Real-time Gesture Animation Generation from Speech for Virtual Human Interaction

We propose a real-time system for synthesizing gestures directly from sp...
research
07/23/2022

Audio-driven Neural Gesture Reenactment with Video Motion Graphs

Human speech is often accompanied by body gestures including arm and han...
research
07/23/2020

Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics

We propose a novel learned deep prior of body motion for 3D hand shape s...
research
03/30/2022

Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation

We propose an approach to estimate arm and hand dynamics from monocular ...
research
10/15/2022

MoRSE: Deep Learning-based Arm Gesture Recognition for Search and Rescue Operations

Efficient and quick remote communication in search and rescue operations...
research
03/10/2022

BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis

Achieving realistic, vivid, and human-like synthesized conversational ge...
research
07/13/2023

Augmented Co-Speech Gesture Generation: Including Form and Meaning Features to Guide Learning-Based Gesture Synthesis

Due to their significance in human communication, the automatic generati...

Please sign up or login with your details

Forgot password? Click here to reset