Flat latent manifolds for music improvisation between human and machine

02/23/2022
by   Nutan Chen, et al.
0

The use of machine learning in artistic music generation leads to controversial discussions of the quality of art, for which objective quantification is nonsensical. We therefore consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal improvisation is to lead to new experiences, both for the musician and the audience. To obtain this behaviour, we resort to the framework of recurrent Variational Auto-Encoders (VAE) and learn to generate music, seeded by a human musician. In the learned model, we generate novel musical sequences by interpolation in latent space. Standard VAEs however do not guarantee any form of smoothness in their latent representation. This translates into abrupt changes in the generated music sequences. To overcome these limitations, we regularise the decoder and endow the latent space with a flat Riemannian manifold, i.e., a manifold that is isometric to the Euclidean space. As a result, linearly interpolating in the latent space yields realistic and smooth musical changes that fit the type of machine–musician interactions we aim for. We provide empirical evidence for our method via a set of experiments on music datasets and we deploy our model for an interactive jam session with a professional drummer. The live performance provides qualitative evidence that the latent representation can be intuitively interpreted and exploited by the drummer to drive the interplay. Beyond the musical application, our approach showcases an instance of human-centred design of machine-learning models, driven by interpretability and the interaction with the end user.

READ FULL TEXT

page 3

page 5

page 6

page 7

page 11

page 12

page 13

page 16

research
06/03/2021

LyricJam: A system for generating lyrics for live instrumental music

We describe a real-time system that receives a live audio stream from a ...
research
08/17/2020

PIANOTREE VAE: Structured Representation Learning for Polyphonic Music

The dominant approach for music representation learning involves the dee...
research
01/15/2020

Learning a Latent Space of Style-Aware Symbolic Music Representations by Adversarial Autoencoders

We address the challenging open problem of learning an effective latent ...
research
02/12/2020

Learning Flat Latent Manifolds with VAEs

Measuring the similarity between data points often requires domain knowl...
research
01/15/2020

Learning Style-Aware Symbolic Music Representations by Adversarial Autoencoders

We address the challenging open problem of learning an effective latent ...
research
06/01/2018

Learning a Latent Space of Multitrack Measures

Discovering and exploring the underlying structure of multi-instrumental...
research
08/10/2023

Exploring XAI for the Arts: Explaining Latent Space in Generative Music

Explainable AI has the potential to support more interactive and fluid c...

Please sign up or login with your details

Forgot password? Click here to reset