Real-Time Lip Sync for Live 2D Animation

by   Deepali Aneja, et al.

The emergence of commercial tools for real-time performance-based 2D animation has enabled 2D characters to appear on live broadcasts and streaming platforms. A key requirement for live animation is fast and accurate lip sync that allows characters to respond naturally to other actors or the audience through the voice of a human performer. In this work, we present a deep learning based interactive system that automatically generates live lip sync for layered 2D characters using a Long Short Term Memory (LSTM) model. Our system takes streaming audio as input and produces viseme sequences with less than 200ms of latency (including processing time). Our contributions include specific design decisions for our feature definition and LSTM configuration that provide a small but useful amount of lookahead to produce accurate lip sync. We also describe a data augmentation procedure that allows us to achieve good results with a very small amount of hand-animated training data (13-20 minutes). Extensive human judgement experiments show that our results are preferred over several competing methods, including those that only support offline (non-live) processing. Video summary and supplementary results at GitHub link:


page 2

page 4

page 5

page 6

page 7

page 9

page 10

page 11


Fast Online "Next Best Offers" using Deep Learning

In this paper, we present iPrescribe, a scalable low-latency architectur...

French Word Recognition through a Quick Survey on Recurrent Neural Networks Using Long-Short Term Memory RNN-LSTM

Optical character recognition (OCR) is a fundamental problem in computer...

A Scalable Framework for Multilevel Streaming Data Analytics using Deep Learning

The rapid growth of data in velocity, volume, value, variety, and veraci...

DeepAuto: A Hierarchical Deep Learning Framework for Real-Time Prediction in Cellular Networks

Accurate real-time forecasting of key performance indicators (KPIs) is a...

Interpretable Real-Time Win Prediction for Honor of Kings, a Popular Mobile MOBA Esport

With the rapid prevalence and explosive development of MOBA esports (Mul...

ImageBox3: No-Server Tile Serving to Traverse Whole Slide Images on the Web

Whole slide imaging (WSI) has become the primary modality for digital pa...

Code Repositories


Real-Time Lip Sync for Live 2D Animation

view repo

Please sign up or login with your details

Forgot password? Click here to reset