Seeing Through Noise: Visually Driven Speaker Separation and Enhancement

08/22/2017
by   Aviv Gabbay, et al.
0

Isolating the voice of a specific person while filtering out other voices or background noises is challenging when video is shot in noisy environments, using a single microphone. For example, video conferences from home or office are disturbed by other voices, TV reporting from city streets is mixed with traffic noise, etc. We propose audio-visual methods to isolate the voice of a single speaker and eliminate unrelated sounds. Face motions captured in the video are used to estimate the speaker's voice, which is applied as a filter on the input audio. This approach avoids using mixtures of sounds in the learning process, as the number of such possible mixtures is huge, and would inevitably bias the trained model.

READ FULL TEXT
research
11/23/2017

Visual Speech Enhancement

When video is shot in noisy environment, the voice of a speaker seen in ...
research
11/23/2017

Visual Speech Enhancement using Noise-Invariant Training

Visual speech enhancement is used on videos shot in noisy environments t...
research
07/11/2019

My lips are concealed: Audio-visual speech enhancement through obstructions

Our objective is an audio-visual model for separating a single speaker f...
research
08/10/2020

Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training

Data efficient voice cloning aims at synthesizing target speaker's voice...
research
03/23/2023

Better Together: Dialogue Separation and Voice Activity Detection for Audio Personalization in TV

In TV services, dialogue level personalization is key to meeting user pr...
research
05/17/2020

Multimodal Target Speech Separation with Voice and Face References

Target speech separation refers to isolating target speech from a multi-...
research
07/20/2018

A Fully Convolutional Neural Network Approach to End-to-End Speech Enhancement

This paper will describe a novel approach to the cocktail party problem ...

Please sign up or login with your details

Forgot password? Click here to reset