DeepAI
Log In Sign Up

Move2Hear: Active Audio-Visual Source Separation

05/15/2021
by   Sagnik Majumder, et al.
6

We introduce the active audio-visual source separation problem, where an agent must move intelligently in order to better isolate the sounds coming from an object of interest in its environment. The agent hears multiple audio sources simultaneously (e.g., a person speaking down the hall in a noisy household) and must use its eyes and ears to automatically separate out the sounds originating from the target object within a limited time budget. Towards this goal, we introduce a reinforcement learning approach that trains movement policies controlling the agent's camera and microphone placement over time, guided by the improvement in predicted audio separation quality. We demonstrate our approach in scenarios motivated by both augmented reality (system is already co-located with the target object) and mobile robotics (agent begins arbitrarily far from the target object). Using state-of-the-art realistic audio-visual simulations in 3D environments, we demonstrate our model's ability to find minimal movement sequences with maximal payoff for audio source separation. Project: http://vision.cs.utexas.edu/projects/move2hear.

READ FULL TEXT

page 1

page 9

page 16

02/02/2022

Active Audio-Visual Separation of Dynamic Sound Sources

We explore active audio-visual separation for dynamic sound sources, whe...
04/16/2019

Co-Separating Sounds of Visual Objects

Learning how objects sound from video is challenging, since they often h...
09/24/2021

Visual Scene Graphs for Audio Source Separation

State-of-the-art approaches for visually-guided audio source separation ...
04/18/2019

Self-Supervised Audio-Visual Co-Segmentation

Segmenting objects in images and separating sound sources in audio are c...
07/20/2020

Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation

Stereophonic audio is an indispensable ingredient to enhance human audit...
06/08/2022

Few-Shot Audio-Visual Learning of Environment Acoustics

Room impulse response (RIR) functions capture how the surrounding physic...
03/18/2022

RoSS: Utilizing Robotic Rotation for Audio Source Separation

This paper considers the problem of audio source separation where the go...