Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

03/29/2023
by   Kun Su, et al.
0

Modeling sounds emitted from physical object interactions is critical for immersive perceptual experiences in real and virtual worlds. Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound. However, they require fine details of both the object geometries and impact locations, which are rarely available in the real world and can not be applied to synthesize impact sounds from common videos. On the other hand, existing video-driven deep learning-based approaches could only capture the weak correspondence between visual content and impact sounds since they lack of physics knowledge. In this work, we propose a physics-driven diffusion model that can synthesize high-fidelity impact sound for a silent video clip. In addition to the video content, we propose to use additional physics priors to guide the impact sound synthesis procedure. The physics priors include both physics parameters that are directly estimated from noisy real-world impact sound examples without sophisticated setup and learned residual parameters that interpret the sound environment via neural networks. We further implement a novel diffusion model with specific training and inference strategies to combine physics priors and visual information for impact sound synthesis. Experimental results show that our model outperforms several existing systems in generating realistic impact sounds. More importantly, the physics-based representations are fully interpretable and transparent, thus enabling us to perform sound editing flexibly.

READ FULL TEXT

page 7

page 12

research
06/29/2017

Toward Inverse Control of Physics-Based Sound Synthesis

Long Short-Term Memory networks (LSTMs) can be trained to realize invers...
research
11/06/2022

Towards Real World HDRTV Reconstruction: A Data Synthesis-based Approach

Existing deep learning based HDRTV reconstruction methods assume one kin...
research
03/25/2019

Physics-based Neural Networks for Shape from Polarization

How should prior knowledge from physics inform a neural network solution...
research
12/28/2015

Visually Indicated Sounds

Objects make distinctive sounds when they are hit or scratched. These so...
research
12/03/2022

Impact of physical model error on State Estimation for neutronics applications

In this paper, we consider the inverse problem of state estimation of nu...
research
05/15/2018

A nonlinear and time-dependent visco-elasto-plastic rheology model for studying shock-physics phenomena

We present a simple and efficient implementation of a viscous creep rheo...
research
04/17/2023

Unleashing the Power of Sound: Revisiting the Physics of Notations for Modelling with auditory symbols

Sound - the oft-neglected sense for Software Engineering - is a crucial ...

Please sign up or login with your details

Forgot password? Click here to reset