DeepAI AI Chat
Log In Sign Up

Spectrogram Inpainting for Interactive Generation of Instrument Sounds

by   Théis Bazin, et al.

Modern approaches to sound synthesis using deep neural networks are hard to control, especially when fine-grained conditioning information is not available, hindering their adoption by musicians. In this paper, we cast the generation of individual instrumental notes as an inpainting-based task, introducing novel and unique ways to iteratively shape sounds. To this end, we propose a two-step approach: first, we adapt the VQ-VAE-2 image generation architecture to spectrograms in order to convert real-valued spectrograms into compact discrete codemaps, we then implement token-masked Transformers for the inpainting-based generation of these codemaps. We apply the proposed architecture on the NSynth dataset on masked resampling tasks. Most crucially, we open-source an interactive web interface to transform sounds by inpainting, for artists and practitioners alike, opening up to new, creative uses.


page 7

page 8


NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting

Inpainting-based generative modeling allows for stimulating human-machin...

StructureFlow: Image Inpainting via Structure-aware Appearance Flow

Image inpainting techniques have shown significant improvements by using...

Real-valued parametric conditioning of an RNN for interactive sound synthesis

A Recurrent Neural Network (RNN) for audio synthesis is trained by augme...

Compatible and Diverse Fashion Image Inpainting

Visual compatibility is critical for fashion analysis, yet is missing in...

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation

Spatial control is a core capability in controllable image generation. A...

Guidefill: GPU Accelerated, Artist Guided Geometric Inpainting for 3D Conversion

The conversion of traditional film into stereo 3D has become an importan...

The Piano Inpainting Application

Autoregressive models are now capable of generating high-quality minute-...