An improper estimator with optimal excess risk in misspecified density estimation and logistic regression

12/23/2019
by   Jaouad Mourtada, et al.
8

We introduce a procedure for predictive conditional density estimation under logarithmic loss, which we call SMP (Sample Minmax Predictor). This predictor minimizes a new general excess risk bound, which critically remains valid under model misspecification. On standard examples, this bound scales as d/n where d is the dimension of the model and n the sample size, regardless of the true distribution. The SMP, which is an improper (out-of-model) procedure, improves over proper (within-model) estimators (such as the maximum likelihood estimator), whose excess risk can degrade arbitrarily in the misspecified case. For density estimation, our bounds improve over approaches based on online-to-batch conversion, by removing suboptimal log n factors, addressing an open problem from Grünwald and Kotłowski (2011) for the considered models. For the Gaussian linear model, the SMP admits an explicit expression, and its expected excess risk in the general misspecified case is at most twice the minimax excess risk in the well-specified case, but without any condition on the noise variance or approximation error of the linear model. For logistic regression, a penalized SMP can be computed efficiently by training two logistic regressions, and achieves a non-asymptotic excess risk of O((d + B^2R^2)/n), where R is a bound on the norm of the features and B the norm of the comparison linear predictor. This improves the rates of proper (within-model) estimators, since such procedures can achieve no better rate than min(BR/√(n),de^BR/n) in general. This also provides a computationally more efficient alternative to approaches based on online-to-batch conversion of Bayesian mixture procedures, which require approximate posterior sampling, thereby partly answering a question by Foster et al. (2018).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/25/2018

Logistic Regression: The Importance of Being Improper

Learning linear predictors with the logistic loss---both in stochastic a...
research
10/06/2021

Efficient Methods for Online Multiclass Logistic Regression

Multiclass logistic regression is a fundamental task in machine learning...
research
03/15/2012

A Family of Computationally Efficient and Simple Estimators for Unnormalized Statistical Models

We introduce a new family of estimators for unnormalized statistical mod...
research
05/25/2023

Finite sample rates for logistic regression with small noise or few samples

The logistic regression estimator is known to inflate the magnitude of i...
research
05/16/2019

Adaptive estimation in the linear random coefficients model when regressors have limited variation

We consider a linear model where the coefficients-intercept and slopes-a...
research
09/21/2021

Minimax Rates for Conditional Density Estimation via Empirical Entropy

We consider the task of estimating a conditional density using i.i.d. sa...
research
10/23/2016

Online Classification with Complex Metrics

We present a framework and analysis of consistent binary classification ...

Please sign up or login with your details

Forgot password? Click here to reset