Performance of Bayesian linear regression in a model with mismatch

07/14/2021
by   Jean Barbier, et al.
0

For a model of high-dimensional linear regression with random design, we analyze the performance of an estimator given by the mean of a log-concave Bayesian posterior distribution with gaussian prior. The model is mismatched in the following sense: like the model assumed by the statistician, the labels-generating process is linear in the input data, but both the classifier ground-truth prior and gaussian noise variance are unknown to her. This inference model can be rephrased as a version of the Gardner model in spin glasses and, using the cavity method, we provide fixed point equations for various overlap order parameters, yielding in particular an expression for the mean-square reconstruction error on the classifier (under an assumption of uniqueness of solutions). As a direct corollary we obtain an expression for the free energy. Similar models have already been studied by Shcherbina and Tirozzi and by Talagrand, but our arguments are more straightforward and some assumptions are relaxed. An interesting consequence of our analysis is that in the random design setting of ridge regression, the performance of the posterior mean is independent of the noise variance (or "temperature") assumed by the statistician, and matches the one of the usual (zero temperature) ridge estimator.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/25/2017

On estimation of the noise variance in high-dimensional linear models

We consider the problem of recovering the unknown noise variance in the ...
research
10/30/2022

Robust and Tuning-Free Sparse Linear Regression via Square-Root Slope

We consider the high-dimensional linear regression model and assume that...
research
03/14/2022

The TAP free energy for high-dimensional linear regression

We derive a variational representation for the log-normalizing constant ...
research
08/30/2020

Bayesian High-dimensional Semi-parametric Inference beyond sub-Gaussian Errors

We consider a sparse linear regression model with unknown symmetric erro...
research
09/11/2020

On polynomial-time computation of high-dimensional posterior measures by Langevin-type algorithms

The problem of generating random samples of high-dimensional posterior d...
research
09/20/2018

Admissibility of the usual confidence set for the mean of a univariate or bivariate normal population: The unknown-variance case

In the Gaussian linear regression model (with unknown mean and variance)...
research
11/27/2020

Learning to extrapolate using continued fractions: Predicting the critical temperature of superconductor materials

In Artificial Intelligence we often seek to identify an unknown target f...

Please sign up or login with your details

Forgot password? Click here to reset