Video Background Music Generation with Controllable Music Transformer

11/16/2021
by   Shangzhe Di, et al.
0

In this work, we address the task of video background music generation. Some previous works achieve effective music generation but are unable to generate melodious music tailored to a particular video, and none of them considers the video-music rhythmic consistency. To generate the background music that matches the given video, we first establish the rhythmic relations between video and background music. In particular, we connect timing, motion speed, and motion saliency from video with beat, simu-note density, and simu-note strength from music, respectively. We then propose CMT, a Controllable Music Transformer that enables local control of the aforementioned rhythmic features and global control of the music genre and instruments. Objective and subjective evaluations show that the generated background music has achieved satisfactory compatibility with the input videos, and at the same time, impressive music quality. Code and models are available at https://github.com/wzk1015/video-bgm-generation.

READ FULL TEXT
research
01/22/2023

Dance2MIDI: Dance-driven multi-instruments music generation

Dance-driven music generation aims to generate musical pieces conditione...
research
11/21/2022

Video Background Music Generation: Dataset, Method and Evaluation

Music is essential when editing videos, but selecting music manually is ...
research
06/08/2023

Simple and Controllable Music Generation

We tackle the task of conditional music generation. We introduce MusicGe...
research
02/17/2022

Chord-Conditioned Melody Choralization with Controllable Harmonicity and Polyphonicity

Melody choralization, i.e. generating a four-part chorale based on a use...
research
05/05/2023

Exploring Softly Masked Language Modelling for Controllable Symbolic Music Generation

This document presents some early explorations of applying Softly Masked...
research
03/18/2021

DanceNet3D: Music Based Dance Generation with Parametric Motion Transformer

In this work, we propose a novel deep learning framework that can genera...
research
08/18/2022

Musika! Fast Infinite Waveform Music Generation

Fast and user-controllable music generation could enable novel ways of c...

Please sign up or login with your details

Forgot password? Click here to reset