Spectrogram Inpainting for Interactive Generation of Instrument Sounds

04/15/2021
by   Théis Bazin, et al.
0

Modern approaches to sound synthesis using deep neural networks are hard to control, especially when fine-grained conditioning information is not available, hindering their adoption by musicians. In this paper, we cast the generation of individual instrumental notes as an inpainting-based task, introducing novel and unique ways to iteratively shape sounds. To this end, we propose a two-step approach: first, we adapt the VQ-VAE-2 image generation architecture to spectrograms in order to convert real-valued spectrograms into compact discrete codemaps, we then implement token-masked Transformers for the inpainting-based generation of these codemaps. We apply the proposed architecture on the NSynth dataset on masked resampling tasks. Most crucially, we open-source an interactive web interface to transform sounds by inpainting, for artists and practitioners alike, opening up to new, creative uses.

READ FULL TEXT

page 7

page 8

research
07/23/2019

NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting

Inpainting-based generative modeling allows for stimulating human-machin...
research
08/11/2019

StructureFlow: Image Inpainting via Structure-aware Appearance Flow

Image inpainting techniques have shown significant improvements by using...
research
07/01/2023

WavePaint: Resource-efficient Token-mixer for Self-supervised Inpainting

Image inpainting, which refers to the synthesis of missing regions in an...
research
05/28/2018

Real-valued parametric conditioning of an RNN for interactive sound synthesis

A Recurrent Neural Network (RNN) for audio synthesis is trained by augme...
research
02/04/2019

Compatible and Diverse Fashion Image Inpainting

Visual compatibility is critical for fashion analysis, yet is missing in...
research
11/16/2016

Guidefill: GPU Accelerated, Artist Guided Geometric Inpainting for 3D Conversion

The conversion of traditional film into stereo 3D has become an importan...
research
04/13/2023

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation

Spatial control is a core capability in controllable image generation. A...

Please sign up or login with your details

Forgot password? Click here to reset