DeepAI AI Chat
Log In Sign Up

Visual to Sound: Generating Natural Sound for Videos in the Wild

by   Yipin Zhou, et al.

As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world. Often correlated during natural events, these two modalities combine to jointly affect human perception. In this paper, we pose the task of generating sound given visual input. Such capabilities could help enable applications in virtual reality (generating sound for virtual scenes automatically) or provide additional accessibility to images or videos for people with visual impairments. As a first step in this direction, we apply learning-based methods to generate raw waveform samples given input video frames. We evaluate our models on a dataset of videos containing a variety of sounds (such as ambient sounds and sounds from people/animals). Our experiments show that the generated sounds are fairly realistic and have good temporal synchronization with the visual inputs.


page 3

page 7


Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality

Ambisonics i.e., a full-sphere surround sound, is quintessential with 36...

El Sonido como Elemento Clave en Prácticas de Realidad Virtual

This article discusses the importance of sound for virtual reality syste...

Visually Indicated Sounds

Objects make distinctive sounds when they are hit or scratched. These so...

VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement

Video to sound generation aims to generate realistic and natural sound g...

Improving Visual Recognition using Ambient Sound for Supervision

Our brains combine vision and hearing to create a more elaborate interpr...

BatVision: Learning to See 3D Spatial Layout with Two Ears

Virtual camera images showing the correct layout of a space ahead can be...

Generating Visually Aligned Sound from Videos

We focus on the task of generating sound from natural videos, and the so...