Log In Sign Up

DualLip: A System for Joint Lip Reading and Generation

by   Weicong Chen, et al.

Lip reading aims to recognize text from talking lip, while lip generation aims to synthesize talking lip according to text, which is a key component in talking face generation and is a dual task of lip reading. In this paper, we develop DualLip, a system that jointly improves lip reading and generation by leveraging the task duality and using unlabeled text and lip video data. The key ideas of the DualLip include: 1) Generate lip video from unlabeled text with a lip generation model, and use the pseudo pairs to improve lip reading; 2) Generate text from unlabeled lip video with a lip reading model, and use the pseudo pairs to improve lip generation. We further extend DualLip to talking face generation with two additionally introduced components: lip to face generation and text to speech generation. Experiments on GRID and TCD-TIMIT demonstrate the effectiveness of DualLip on improving lip reading, lip generation, and talking face generation by utilizing unlabeled data. Specifically, the lip generation model in our DualLip system trained with only10 paired data. And on the GRID benchmark of lip reading, we achieve 1.16 character error rate and 2.71 state-of-the-art models using the same amount of paired data.


Lip reading using external viseme decoding

Lip-reading is the operation of recognizing speech from lip movements. T...

GuidedMix-Net: Learning to Improve Pseudo Masks Using Labeled Images as Reference

Semi-supervised learning is a challenging problem which aims to construc...

Deep Lip Reading: a comparison of models and an online application

The goal of this paper is to develop state-of-the-art models for lip rea...

Confidence-aware Non-repetitive Multimodal Transformers for TextCaps

When describing an image, reading text in the visual scene is crucial to...

Revisiting Self-Training for Neural Sequence Generation

Self-training is one of the earliest and simplest semi-supervised method...

AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style

While recent text to speech (TTS) models perform very well in synthesizi...

Field Extraction from Forms with Unlabeled Data

We propose a novel framework to conduct field extraction from forms with...