A proof of consistency and model-selection optimality on the empirical Bayes method

05/26/2022
by   Dye SK Sato, et al.
0

We study the consistency and optimality of the maximum marginal likelihood estimate (MMLE) in the hyperparameter inference for large-degree-of-freedom models. We perform main analyses within the exponential family, where the natural parameters are hyperparameters. First, we prove the consistency of the MMLE for the general linear models when estimating the scales of variance in the likelihood and prior. The proof is independent of the number ratio of data to model parameters and excepts the ill-posedness of the associated regularized least-square model-parameter estimate that is shown asymptotically unbiased. Second, we generalize the proof to other models with a finite number of hyperparameters. We find that the extensive properties of cost functions in the exponential family generally yield the consistency of the MMLE for the likelihood hyperparameters. Besides, we show the MMLE asymptotically almost surely minimizes the Kullback-Leibler divergence between the prior and true predictive distributions even if the true data distribution is outside the model space under the hypothetical asymptotic normality of the predictive distributions applicable to non-exponential model families. Our proof validates the empirical Bayes method using the hyperparameter MMLE in the asymptotics of many model parameters, ensuring the same qualification for the empirical-cross-entropy cross-validation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/27/2015

Bayesian Cross Validation and WAIC for Predictive Prior Design in Regular Asymptotic Theory

Prior design is one of the most important problems in both statistics an...
research
02/07/2020

Empirical Bayes for Large-scale Randomized Experiments: a Spectral Approach

Large-scale randomized experiments, sometimes called A/B tests, are incr...
research
10/27/2019

Prior specification via prior predictive matching: Poisson matrix factorization and beyond

Hyperparameter optimization for machine learning models is typically car...
research
08/27/2018

Empirical likelihood for linear models with spatial errors

For linear models with spatial errors, the empirical likelihood ratio st...
research
04/08/2023

Inference on a class of exponential families on permutations

In this paper we study a class of exponential family on permutations, wh...
research
07/10/2012

Dual-Space Analysis of the Sparse Linear Model

Sparse linear (or generalized linear) models combine a standard likeliho...
research
11/03/2020

Robust estimation of a regression function in exponential families

We observe n pairs X_1=(W_1,Y_1),…,X_n=(W_n,Y_n) of independent random v...

Please sign up or login with your details

Forgot password? Click here to reset