Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models

by   Emiel Hoogeboom, et al.

The field of language modelling has been largely dominated by autoregressive models, for which sampling is inherently difficult to parallelize. This paper introduces two new classes of generative models for categorical data such as language or image segmentation: Argmax Flows and Multinomial Diffusion. Argmax Flows are defined by a composition of a continuous distribution (such as a normalizing flow), and an argmax function. To optimize this model, we learn a probabilistic inverse for the argmax that lifts the categorical data to a continuous space. Multinomial Diffusion gradually adds categorical noise in a diffusion process, for which the generative denoising process is learned. We demonstrate that our models perform competitively on language modelling and modelling of image segmentation maps.



page 8


Categorical Normalizing Flows via Continuous Transformations

Despite their popularity, to date, the application of normalizing flows ...

Diffusion Models for Implicit Image Segmentation Ensembles

Diffusion models have shown impressive performance for generative modell...

Equivariant Diffusion for Molecule Generation in 3D

This work introduces a diffusion model for molecule generation in 3D tha...

Diffusion models as plug-and-play priors

We consider the problem of inferring high-dimensional data 𝐱 in a model ...

Symbolic Music Generation with Diffusion Models

Score-based generative models and diffusion probabilistic models have be...

Variational Diffusion Models

Diffusion-based generative models have demonstrated a capacity for perce...

Generative Modelling With Inverse Heat Dissipation

While diffusion models have shown great success in image generation, the...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.