On Computationally Efficient Learning of Exponential Family Distributions

by   Abhin Shah, et al.

We consider the classical problem of learning, with arbitrary accuracy, the natural parameters of a k-parameter truncated minimal exponential family from i.i.d. samples in a computationally and statistically efficient manner. We focus on the setting where the support as well as the natural parameters are appropriately bounded. While the traditional maximum likelihood estimator for this class of exponential family is consistent, asymptotically normal, and asymptotically efficient, evaluating it is computationally hard. In this work, we propose a novel loss function and a computationally efficient estimator that is consistent as well as asymptotically normal under mild conditions. We show that, at the population level, our method can be viewed as the maximum likelihood estimation of a re-parameterized distribution belonging to the same class of exponential family. Further, we show that our estimator can be interpreted as a solution to minimizing a particular Bregman score as well as an instance of minimizing the surrogate likelihood. We also provide finite sample guarantees to achieve an error (in ℓ_2-norm) of α in the parameter estimation with sample complexity O( poly(k)/α^2). Our method achives the order-optimal sample complexity of O( log(k)/α^2) when tailored for node-wise-sparse Markov random fields. Finally, we demonstrate the performance of our estimator via numerical experiments.


page 1

page 2

page 3

page 4


A Computationally Efficient Method for Learning Exponential Family Distributions

We consider the question of learning the natural parameters of a k param...

Diagnostic Tool for Out-of-Sample Model Evaluation

Assessment of model fitness is an important step in many problems. Model...

Inference on a class of exponential families on permutations

In this paper we study a class of exponential family on permutations, wh...

On Learning Continuous Pairwise Markov Random Fields

We consider learning a sparse pairwise Markov Random Field (MRF) with co...

Provable benefits of score matching

Score matching is an alternative to maximum likelihood (ML) for estimati...

Computationally efficient likelihood inference in exponential families when the maximum likelihood estimator does not exist

In a regular full exponential family, the maximum likelihood estimator (...

A parallel algorithm for penalized learning of the multivariate exponential family from data of mixed types

Computational efficient evaluation of penalized estimators of multivaria...

Please sign up or login with your details

Forgot password? Click here to reset