A Characterization of Mean Squared Error for Estimator with Bagging

by   Martin Mihelich, et al.

Bagging can significantly improve the generalization performance of unstable machine learning algorithms such as trees or neural networks. Though bagging is now widely used in practice and many empirical studies have explored its behavior, we still know little about the theoretical properties of bagged predictions. In this paper, we theoretically investigate how the bagging method can reduce the Mean Squared Error (MSE) when applied on a statistical estimator. First, we prove that for any estimator, increasing the number of bagged estimators N in the average can only reduce the MSE. This intuitive result, observed empirically and discussed in the literature, has not yet been rigorously proved. Second, we focus on the standard estimator of variance called unbiased sample variance and we develop an exact analytical expression of the MSE for this estimator with bagging. This allows us to rigorously discuss the number of iterations N and the batch size m of the bagging method. From this expression, we state that only if the kurtosis of the distribution is greater than 3/2, the MSE of the variance estimator can be reduced with bagging. This result is important because it demonstrates that for distribution with low kurtosis, bagging can only deteriorate the performance of a statistical prediction. Finally, we propose a novel general-purpose algorithm to estimate with high precision the variance of a sample.



There are no comments yet.


page 1

page 2

page 3

page 4


Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator

Gradient estimation in models with discrete latent variables is a challe...

Convergence of Chao Unseen Species Estimator

Support size estimation and the related problem of unseen species estima...

Estimation of Measures for Two-Way Contingency Tables Using the Bayesian Estimators

In the analysis of two-way contingency tables, the measures for represen...

Private Protocols for U-Statistics in the Local Model and Beyond

In this paper, we study the problem of computing U-statistics of degree ...

Estimation and Inference with Trees and Forests in High Dimensions

We analyze the finite sample mean squared error (MSE) performance of reg...

Optimal Estimation with Complete Subsets of Instruments

In this paper we propose a two-stage least squares (2SLS) estimator whos...

Mean Estimation from One-Bit Measurements

We consider the problem of estimating the mean of a symmetric log-concav...

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.