SongDriver2: Real-time Emotion-based Music Arrangement with Soft Transition

05/14/2023
by   Zihao Wang, et al.
0

Real-time emotion-based music arrangement, which aims to transform a given music piece into another one that evokes specific emotional resonance with the user in real-time, holds significant application value in various scenarios, e.g., music therapy, video game soundtracks, and movie scores. However, balancing emotion real-time fit with soft emotion transition is a challenge due to the fine-grained and mutable nature of the target emotion. Existing studies mainly focus on achieving emotion real-time fit, while the issue of soft transition remains understudied, affecting the overall emotional coherence of the music. In this paper, we propose SongDriver2 to address this balance. Specifically, we first recognize the last timestep's music emotion and then fuse it with the current timestep's target input emotion. The fused emotion then serves as the guidance for SongDriver2 to generate the upcoming music based on the input melody data. To adjust music similarity and emotion real-time fit flexibly, we downsample the original melody and feed it into the generation model. Furthermore, we design four music theory features to leverage domain knowledge to enhance emotion information and employ semi-supervised learning to mitigate the subjective bias introduced by manual dataset annotation. According to the evaluation results, SongDriver2 surpasses the state-of-the-art methods in both objective and subjective metrics. These results demonstrate that SongDriver2 achieves real-time fit and soft transitions simultaneously, enhancing the coherence of the generated music.

READ FULL TEXT
research
12/16/2021

EmotionBox: a music-element-driven emotional music generation system using Recurrent Neural Network

With the development of deep neural networks, automatic music compositio...
research
03/03/2023

Decoding and Visualising Intended Emotion in an Expressive Piano Performance

Expert musicians can mould a musical piece to convey specific emotions t...
research
09/13/2022

SongDriver: Real-time Music Accompaniment Generation without Logical Latency nor Exposure Bias

Real-time music accompaniment generation has a wide range of application...
research
08/28/2023

Symbolic Acoustic: Multi-domain Music Emotion Modeling for Instrumental Music

Music Emotion Recognition involves the automatic identification of emoti...
research
12/31/2021

Evaluating Deep Music Generation Methods Using Data Augmentation

Despite advances in deep algorithmic music generation, evaluation of gen...
research
07/02/2019

Adaptive Music Composition for Games

The generation of music that adapts dynamically to content and actions h...
research
09/17/2017

A Categorical Approach for Recognizing Emotional Effects of Music

Recently, digital music libraries have been developed and can be plainly...

Please sign up or login with your details

Forgot password? Click here to reset