Exponential Family Estimation via Adversarial Dynamics Embedding

04/27/2019
by   Bo Dai, et al.
28

We present an efficient algorithm for maximum likelihood estimation (MLE) of the general exponential family, even in cases when the energy function is represented by a deep neural network. We consider the primal-dual view of the MLE for the kinectics augmented model, which naturally introduces an adversarial dual sampler. The sampler will be represented by a novel neural network architectures, dynamics embeddings, mimicking the dynamical-based samplers, e.g., Hamiltonian Monte-Carlo and its variants. The dynamics embedding parametrization inherits the flexibility from HMC, and provides tractable entropy estimation of the augmented model. Meanwhile, it couples the adversarial dual samplers with the primal model, reducing memory and sample complexity. We further show that several existing estimators, including contrastive divergence (Hinton, 2002), score matching (Hyvärinen, 2005), pseudo-likelihood (Besag, 1975), noise-contrastive estimation (Gutmann and Hyvärinen, 2010), non-local contrastive objectives (Vickrey et al., 2010), and minimum probability flow (Sohl-Dickstein et al., 2011), can be recast as the special cases of the proposed method with different prefixed dual samplers. Finally, we empirically demonstrate the superiority of the proposed estimator against existing state-of-the-art methods on synthetic and real-world benchmarks.

READ FULL TEXT

page 12

page 14

page 25

page 26

research
11/06/2018

Kernel Exponential Family Estimation via Doubly Dual Embedding

We investigate penalized maximum log-likelihood estimation for exponenti...
research
08/24/2018

Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance

There are many models, often called unnormalized models, whose normalizi...
research
03/06/2023

Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems

The entropic fictitious play (EFP) is a recently proposed algorithm that...
research
09/11/2020

A simulation study of semiparametric estimation in copula models based on minimum Alpha-Divergence

The purpose of this paper is to introduce two semiparametric methods for...
research
09/22/2022

Approximating (p,2) flexible graph connectivity via the primal-dual method

We consider the Flexible Graph Connectivity model (denoted FGC) introduc...
research
05/19/2016

Recurrent Exponential-Family Harmoniums without Backprop-Through-Time

Exponential-family harmoniums (EFHs), which extend restricted Boltzmann ...
research
04/07/2023

Conservative objective models are a special kind of contrastive divergence-based energy model

In this work we theoretically show that conservative objective models (C...

Please sign up or login with your details

Forgot password? Click here to reset