Finite samples inference and critical dimension for stochastically linear models

by   Vladimir Spokoiny, et al.

The aim of this note is to state a couple of general results about the properties of the penalized maximum likelihood estimators (pMLE) and of the posterior distribution for parametric models in a non-asymptotic setup and for possibly large or even infinite parameter dimension. We consider a special class of stochastically linear smooth (SLS) models satisfying two major conditions: the stochastic component of the log-likelihood is linear in the model parameter, while the expected log-likelihood is a smooth function. The main results simplify a lot if the expected log-likelihood is concave. For the pMLE, we establish a number of finite sample bounds about its concentration and large deviations as well as the Fisher and Wilks expansion. The later results extend the classical asymptotic Fisher and Wilks Theorems about the MLE to the non-asymptotic setup with large parameter dimension which can depend on the sample size. For the posterior distribution, our main result states a Gaussian approximation of the posterior which can be viewed as a finite sample analog of the prominent Bernstein–von Mises Theorem. In all bounds, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension. The results are dimension and coordinate free. In spite of generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. An interesting case of logit regression with smooth or truncation priors is used to specify the results and to explain the main notions.


page 1

page 2

page 3

page 4


Accuracy of Gaussian approximation in nonparametric Bernstein – von Mises Theorem

The prominent Bernstein – von Mises (BvM) result claims that the posteri...

A Diffusion Process Perspective on Posterior Contraction Rates for Parameters

We show that diffusion processes can be exploited to study the posterior...

Nonasymptotic Laplace approximation under model misspecification

We present non-asymptotic two-sided bounds to the log-marginal likelihoo...

Deterministic Inequalities for Smooth M-estimators

Ever since the proof of asymptotic normality of maximum likelihood estim...

Posterior Probabilities: Nonmonotonicity, Asymptotic Rates, Log-Concavity, and Turán's Inequality

In the standard Bayesian framework data are assumed to be generated by a...

On an Asymptotic Distribution for the MLE

The paper presents a novel asymptotic distribution for a mle when the lo...

Concentration of Non-Isotropic Random Tensors with Applications to Learning and Empirical Risk Minimization

Dimension is an inherent bottleneck to some modern learning tasks, where...