Asymptotic Normality and Variance Estimation For Supervised Ensembles

12/02/2019
by   Zhengze Zhou, et al.
0

Ensemble methods based on bootstrapping have improved the predictive accuracy of base learners, but fail to provide a framework in which formal statistical inference can be conducted. Recent theoretical developments suggest taking subsamples without replacement and analyze the resulting estimator in the context of a U-statistic, thus demonstrating asymptotic normality properties. However, we observe that current methods for variance estimation exhibit severe bias when the number of base learners is not large enough, compromising the validity of the resulting confidence intervals or hypothesis tests. This paper shows that similar asymptotics can be achieved by means of V-statistics, corresponding to taking subsamples with replacement. Further, we develop a bias correction algorithm for estimating variance in the limiting distribution, which yields satisfactory results with moderate size of base learners.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/25/2014

Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests

This work develops formal statistical inference procedures for machine l...
research
08/14/2019

Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems

Aggregating multiple learners through an ensemble of models aims to make...
research
06/01/2015

Bootstrap Bias Corrections for Ensemble Methods

This paper examines the use of a residual bootstrap for bias correction ...
research
04/30/2021

Automatic Debiased Machine Learning via Neural Nets for Generalized Linear Regression

We give debiased machine learners of parameters of interest that depend ...
research
02/18/2022

On Variance Estimation of Random Forests

Ensemble methods, such as random forests, are popular in applications du...
research
02/27/2023

U-Statistics for Importance-Weighted Variational Inference

We propose the use of U-statistics to reduce variance for gradient estim...
research
06/09/2022

Diagnosing Ensemble Few-Shot Classifiers

The base learners and labeled samples (shots) in an ensemble few-shot cl...

Please sign up or login with your details

Forgot password? Click here to reset