Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network

11/02/2018
by   Juheon Lee, et al.
0

Automatic choreography generation is a challenging task because it often requires an understanding of two abstract concepts - music and dance - which are realized in the two different modalities, namely audio and video, respectively. In this paper, we propose a music-driven choreography generation system using an auto-regressive encoder-decoder network. To this end, we first collect a set of multimedia clips that include both music and corresponding dance motion. We then extract the joint coordinates of the dancer from video and the mel-spectrogram of music from audio, and train our network using music-choreography pairs as input. Finally, a novel dance motion is generated at the inference time when only music is given as an input. We performed a user study for a qualitative evaluation of the proposed method, and the results show that the proposed model is able to generate musically meaningful and natural dance movements given an unheard song.

READ FULL TEXT

page 2

page 3

research
05/11/2023

V2Meow: Meowing to the Visual Beat via Music Generation

Generating high quality music that complements the visual content of a v...
research
06/25/2021

Transflower: probabilistic autoregressive dance generation with multimodal attention

Dance requires skillful composition of complex movements that follow rhy...
research
11/12/2020

Automatic Neural Lyrics and Melody Composition

In this paper, we propose a technique to address the most challenging as...
research
09/03/2023

MAGMA: Music Aligned Generative Motion Autodecoder

Mapping music to dance is a challenging problem that requires spatial an...
research
04/06/2023

Automatic Detection of Reactions to Music via Earable Sensing

We present GrooveMeter, a novel system that automatically detects vocal ...
research
09/02/2018

PlayNPort: A Portable Wireless Music Player and Text Reader System

Portable Consumer Electronics has made a mark in the industry. With the ...
research
08/22/2023

MusicJam: Visualizing Music Insights via Generated Narrative Illustrations

Visualizing the insights of the invisible music is able to bring listene...

Please sign up or login with your details

Forgot password? Click here to reset