TraDE: Transformers for Density Estimation

04/06/2020
by   Rasool Fakoor, et al.
0

We present TraDE, an attention-based architecture for auto-regressive density estimation. In addition to a Maximum Likelihood loss we employ a Maximum Mean Discrepancy (MMD) two-sample loss to ensure that samples from the estimate resemble the training data. The use of attention means that the model need not retain conditional sufficient statistics during the process beyond what is needed for each covariate. TraDE performs significantly better than existing approaches such differentiable flow based estimators on standard tabular and image-based benchmarks in terms of the log-likelihood on held out data. TraDE works well wide range of tasks that includes classification methods to ascertain the quality of generated samples, out of distribution sample detection, and handling outliers in the training data.

READ FULL TEXT

page 1

page 6

page 7

page 11

research
02/03/2022

Maximum Likelihood Uncertainty Estimation: Robustness to Outliers

We benchmark the robustness of maximum likelihood based uncertainty esti...
research
10/21/2020

Conditional Density Estimation via Weighted Logistic Regressions

Compared to the conditional mean as a simple point estimator, the condit...
research
02/17/2020

On the Discrepancy between Density Estimation and Sequence Generation

Many sequence-to-sequence generation tasks, including machine translatio...
research
05/18/2018

Fast Multivariate Log-Concave Density Estimation

We present a computational approach to log-concave density estimation. T...
research
02/14/2018

Conditional Density Estimation with Bayesian Normalising Flows

Modeling complex conditional distributions is critical in a variety of s...
research
07/21/2019

Noise Regularization for Conditional Density Estimation

Modelling statistical relationships beyond the conditional mean is cruci...

Please sign up or login with your details

Forgot password? Click here to reset