DeepAI AI Chat
Log In Sign Up

Categorical Normalizing Flows via Continuous Transformations

06/17/2020
by   Phillip Lippe, et al.
University of Amsterdam
Google Inc
0

Despite their popularity, to date, the application of normalizing flows on categorical data stays limited. The current practice of using dequantization to map discrete data to a continuous space is inapplicable as categorical data has no intrinsic order. Instead, categorical data have complex and latent relations that must be inferred, like the synonymy between words. In this paper, we investigate Categorical Normalizing Flows, that is normalizing flows for categorical data. By casting the encoding of categorical data in continuous space as a variational inference problem, we jointly optimize the continuous representation and the model likelihood. To maintain unique decoding, we learn a partitioning of the latent space by factorizing the posterior. Meanwhile, the complex relations between the categorical variables are learned by the ensuing normalizing flow, thus maintaining a close-to exact likelihood estimate and making it possible to scale up to a large number of categories. Based on Categorical Normalizing Flows, we propose GraphCNF a permutation-invariant generative model on graphs, outperforming both one-shot and autoregressive flow-based state-of-the-art on molecule generation.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/10/2021

Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models

The field of language modelling has been largely dominated by autoregres...
07/24/2021

Discrete Denoising Flows

Discrete flow-based models are a recently proposed class of generative m...
12/10/2019

Representational Rényi heterogeneity

A discrete system's heterogeneity is measured by the Rényi heterogeneity...
03/15/2022

Categorical Representation Learning and RG flow operators for algorithmic classifiers

Following the earlier formalism of the categorical representation learni...
08/03/2022

AdaCat: Adaptive Categorical Discretization for Autoregressive Models

Autoregressive generative models can estimate complex continuous data di...
03/14/2022

Semi-Discrete Normalizing Flows through Differentiable Tessellation

Mapping between discrete and continuous distributions is a difficult tas...
02/06/2020

Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow

Flow models have recently made great progress at modeling quantized sens...