Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models

02/10/2021
by   Emiel Hoogeboom, et al.
2

The field of language modelling has been largely dominated by autoregressive models, for which sampling is inherently difficult to parallelize. This paper introduces two new classes of generative models for categorical data such as language or image segmentation: Argmax Flows and Multinomial Diffusion. Argmax Flows are defined by a composition of a continuous distribution (such as a normalizing flow), and an argmax function. To optimize this model, we learn a probabilistic inverse for the argmax that lifts the categorical data to a continuous space. Multinomial Diffusion gradually adds categorical noise in a diffusion process, for which the generative denoising process is learned. We demonstrate that our models perform competitively on language modelling and modelling of image segmentation maps.

READ FULL TEXT

Authors

page 8

06/17/2020

Categorical Normalizing Flows via Continuous Transformations

Despite their popularity, to date, the application of normalizing flows ...
12/06/2021

Diffusion Models for Implicit Image Segmentation Ensembles

Diffusion models have shown impressive performance for generative modell...
03/31/2022

Equivariant Diffusion for Molecule Generation in 3D

This work introduces a diffusion model for molecule generation in 3D tha...
06/17/2022

Diffusion models as plug-and-play priors

We consider the problem of inferring high-dimensional data 𝐱 in a model ...
03/30/2021

Symbolic Music Generation with Diffusion Models

Score-based generative models and diffusion probabilistic models have be...
07/01/2021

Variational Diffusion Models

Diffusion-based generative models have demonstrated a capacity for perce...
06/21/2022

Generative Modelling With Inverse Heat Dissipation

While diffusion models have shown great success in image generation, the...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.