DeepAI AI Chat
Log In Sign Up

Robust, scalable and fast bootstrap method for analyzing large scale data

by   Shahab Basiri, et al.

In this paper we address the problem of performing statistical inference for large scale data sets i.e., Big Data. The volume and dimensionality of the data may be so high that it cannot be processed or stored in a single computing node. We propose a scalable, statistically robust and computationally efficient bootstrap method, compatible with distributed processing and storage systems. Bootstrap resamples are constructed with smaller number of distinct data points on multiple disjoint subsets of data, similarly to the bag of little bootstrap method (BLB) [1]. Then significant savings in computation is achieved by avoiding the re-computation of the estimator for each bootstrap sample. Instead, a computationally efficient fixed-point estimation equation is analytically solved via a smart approximation following the Fast and Robust Bootstrap method (FRB) [2]. Our proposed bootstrap method facilitates the use of highly robust statistical methods in analyzing large scale data sets. The favorable statistical properties of the method are established analytically. Numerical examples demonstrate scalability, low complexity and robust statistical performance of the method in analyzing large data sets.


page 1

page 2

page 3

page 4


Two-Stage Robust and Sparse Distributed Statistical Inference for Large-Scale Data

In this paper, we address the problem of conducting statistical inferenc...

Statistical inference in massive datasets by empirical likelihood

In this paper, we propose a new statistical inference method for massive...

The Big Data Bootstrap

The bootstrap provides a simple and powerful means of assessing the qual...

Statistical scalability and approximate inference in distributed computing environments

Harnessing distributed computing environments to build scalable inferenc...

A Cheap Bootstrap Method for Fast Inference

The bootstrap is a versatile inference method that has proven powerful i...

Simultaneous Inference for Massive Data: Distributed Bootstrap

In this paper, we propose a bootstrap method applied to massive data pro...

A Framework for Mediation Analysis with Massive Data

During the past few years, mediation analysis has gained increasing popu...