A Global Bias-Correction DC Method for Biased Estimation under Memory Constraint
share
This paper introduces a global bias-correction divide-and-conquer (GBC-DC) method for biased estimation under the case of memory constraint. In order to introduce the new estimation, a closed representation of the local estimators obtained by the data in each batch is adopted to formulate a pro forma linear regression between the local estimators and the true parameter of interest. A least squares is used within this framework to composite a global estimator of the parameter. Thus, the main difference from the classical DC method is that the new GBC-DC method can absorb the information hidden in the statistical structure and the variables in each batch of data. Consequently, the resulting global estimator is strictly unbiased even if the local estimators have a non-negligible bias. Moreover, the global estimator is consistent under some mild conditions, and even can achieve root-n consistency when the number of batches is large. The new method is simple and computationally efficient, without use of any iterative algorithm and local bias-correction. Moreover, the proposed GBC-DC method applies to various biased estimations such as shrinkage-type estimation and nonparametric regression estimation. Based on our comprehensive simulation studies, the proposed GBC-DC approach is significantly bias-corrected, and the behavior is comparable with that of the full data estimation.
READ FULL TEXT