DeepAI AI Chat
Log In Sign Up

Generalized Resubstitution for Classification Error Estimation

by   Parisa Ghane, et al.
Texas A&M University

We propose the family of generalized resubstitution classifier error estimators based on empirical measures. These error estimators are computationally efficient and do not require re-training of classifiers. The plain resubstitution error estimator corresponds to choosing the standard empirical measure. Other choices of empirical measure lead to bolstered, posterior-probability, Gaussian-process, and Bayesian error estimators; in addition, we propose bolstered posterior-probability error estimators as a new family of generalized resubstitution estimators. In the two-class case, we show that a generalized resubstitution estimator is consistent and asymptotically unbiased, regardless of the distribution of the features and label, if the corresponding generalized empirical measure converges uniformly to the standard empirical measure and the classification rule has a finite VC dimension. A generalized resubstitution estimator typically has hyperparameters that can be tuned to control its bias and variance, which adds flexibility. Numerical experiments with various classification rules trained on synthetic data assess the thefinite-sample performance of several representative generalized resubstitution error estimators. In addition, results of an image classification experiment using the LeNet-5 convolutional neural network and the MNIST data set demonstrate the potential of this class of error estimators in deep learning for computer vision.


Synthetic estimation for the complier average causal effect

We propose an improved estimator of the complier average causal effect (...

Cube root weak convergence of empirical estimators of a density level set

Given n independent random vectors with common density f on ℝ^d, we stud...

Estimation of Measures for Two-Way Contingency Tables Using the Bayesian Estimators

In the analysis of two-way contingency tables, the measures for represen...

On Generalized Schürmann Entropy Estimators

We present a new class of estimators of Shannon entropy for severely und...

Posterior Average Effects

Economists are often interested in computing averages with respect to a ...

Robust Importance Sampling for Error Estimation in the Context of Optimal Bayesian Transfer Learning

Classification has been a major task for building intelligent systems as...