DeepAI AI Chat
Log In Sign Up

A Sequential Addressing Subsampling Method for Massive Data Analysis under Memory Constraint

by   Rui Pan, et al.

The emergence of massive data in recent years brings challenges to automatic statistical inference. This is particularly true if the data are too numerous to be read into memory as a whole. Accordingly, new sampling techniques are needed to sample data from a hard drive. In this paper, we propose a sequential addressing subsampling (SAS) method, that can sample data directly from the hard drive. The newly proposed SAS method is time saving in terms of addressing cost compared to that of the random addressing subsampling (RAS) method. Estimators (e.g., the sample mean) based on the SAS subsamples are constructed, and their properties are studied. We conduct a series of simulation studies to verify the finite sample performance of the proposed SAS estimators. The time cost is also compared between the SAS and RAS methods. An analysis of the airline data is presented for illustration purpose.


page 1

page 2

page 3

page 4


Functional L-Optimality Subsampling for Massive Data

Massive data bring the big challenges of memory and computation for anal...

Statistical inference in massive datasets by empirical likelihood

In this paper, we propose a new statistical inference method for massive...

On the Subbagging Estimation for Massive Data

This article introduces subbagging (subsample aggregating) estimation ap...

Optimal Cox Regression Subsampling Procedure with Rare Events

Massive sized survival datasets are becoming increasingly prevalent with...

Statistical inference for the slope parameter in functional linear regression

In this paper we consider the linear regression model Y =S X+ε with func...

Dynamic statistical inference in massive datastreams

Modern technological advances have expanded the scope of applications re...

Mastering Panel 'Metrics: Causal Impact of Democracy on Growth

The relationship between democracy and economic growth is of long-standi...