Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes

02/02/2023
by   Anton Ratnarajah, et al.
0

We present an end-to-end binaural audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based binaural sound propagation method to generate acoustic effects for 3D models of real environments. Any clean audio or dry audio can be convolved with the generated acoustic effects to render audio corresponding to the real environment. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate acoustic effects from the scene latent vector. Our network is able to handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function to the generator network to incorporate spatial audio effects. Given the source and the listener position, our learning-based binaural sound propagation approach can generate an acoustic effect in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU and can easily handle multiple sources. We have evaluated the accuracy of our approach with binaural acoustic effects generated using an interactive geometric sound propagation algorithm and captured real acoustic effects. We also performed a perceptual evaluation and observed that the audio rendered by our approach is more plausible as compared to audio rendered using prior learning-based sound propagation algorithms.

READ FULL TEXT

page 1

page 4

page 10

page 11

page 12

research
11/14/2019

Scene-Aware Audio Rendering via Deep Acoustic Analysis

We present a new method to capture the acoustic characteristics of real-...
research
09/26/2021

Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse

Interactive audio spatialization technology previously developed for vid...
research
09/16/2019

Real-time 3-D Mapping with Estimating Acoustic Materials

This paper proposes a real-time system integrating an acoustic material ...
research
03/01/2018

Interactive Sound Rendering on Mobile Devices using Ray-Parameterized Reverberation Filters

We present a new sound rendering pipeline that is able to generate plaus...
research
07/24/2021

Dynamic Portal Occlusion for Precomputed Interactive Sound Propagation

An immersive audio-visual experience in games and virtual reality requir...
research
04/26/2021

Points2Sound: From mono to binaural audio using 3D point cloud scenes

Binaural sound that matches the visual counterpart is crucial to bring m...
research
08/09/2023

Sound propagation in realistic interactive 3D scenes with parameterized sources using deep neural operators

We address the challenge of sound propagation simulations in 3D virtual ...

Please sign up or login with your details

Forgot password? Click here to reset