Estimation of Over-parameterized Models via Fitting to Future Observations

06/03/2022
by   Yiran Jiang, et al.
0

From a model-building perspective, in this paper we propose a paradigm shift for fitting over-parameterized models. Philosophically, the mindset is to fit models to future observations rather than to the observed sample. Technically, choosing an imputation model for generating future observations, we fit over-parameterized models to future observations via optimizing an approximation to the desired expected loss-function based on its sample counterpart and an adaptive simplicity-preference function. This technique is discussed in detail to both creating bootstrap imputation and final estimation with bootstrap imputation. The method is illustrated with the many-normal-means problem, n < p linear regression, and deep convolutional neural networks for image classification of MNIST digits. The numerical results demonstrate superior performance across these three different types of applications. For example, for the many-normal-means problem, our method uniformly dominates James-Stein and Efron's g-modeling, and for the MNIST image classification, it performs better than all existing methods and reaches arguably the best possible result. While this paper is largely expository because of the ambitious task of taking a look at over-parameterized models from the new perspective, fundamental theoretical properties are also investigated. We conclude the paper with a few remarks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/19/2023

A New Bootstrap Goodness-of-Fit Test for Normal Linear Regression Models

In this work, the distributional properties of the goodness-of-fit term ...
research
11/29/2018

Accounting for model uncertainty in multiple imputation under complex sampling

Multiple imputation provides an effective way to handle missing data. Wh...
research
04/07/2020

Learning Individual Models for Imputation (Technical Report)

Missing numerical values are prevalent, e.g., owing to unreliable sensor...
research
04/07/2021

Prediction with Missing Data

Missing information is inevitable in real-world data sets. While imputat...
research
10/13/2022

Bootstrap Advantage Estimation for Policy Optimization in Reinforcement Learning

This paper proposes an advantage estimation approach based on data augme...
research
11/25/2020

Modern Multiple Imputation with Functional Data

This work considers the problem of fitting functional models with sparse...

Please sign up or login with your details

Forgot password? Click here to reset