Remixing Music with Visual Conditioning

10/27/2020
by   Li-Chia Yang, et al.
0

We propose a visually conditioned music remixing system by incorporating deep visual and audio models. The method is based on a state of the art audio-visual source separation model which performs music instrument source separation with video information. We modified the model to work with user-selected images instead of videos as visual input during inference to enable separation of audio-only content. Furthermore, we propose a remixing engine that generalizes the task of source separation into music remixing. The proposed method is able to achieve improved audio quality compared to remixing performed by the separate-and-add method with a state-of-the-art audio-visual source separation model.

READ FULL TEXT

page 6

page 7

research
02/03/2021

Music source separation conditioned on 3D point clouds

Recently, significant progress has been made in audio source separation ...
research
07/14/2021

Multi-Task Audio Source Separation

The audio source separation tasks, such as speech enhancement, speech se...
research
04/08/2020

Conditioned Source Separation for Music Instrument Performances

Separating different music instruments playing the same piece is a chall...
research
06/14/2020

Solos: A Dataset for Audio-Visual Music Analysis

In this paper, we present a new dataset of music performance videos whic...
research
02/19/2021

CatNet: music source separation system with mix-audio augmentation

Music source separation (MSS) is the task of separating a music piece in...
research
07/31/2023

DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models

We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framew...
research
05/22/2018

Music Source Separation Using Stacked Hourglass Networks

In this paper, we propose a simple yet effective method for multiple mus...

Please sign up or login with your details

Forgot password? Click here to reset