The biglasso Package: A Memory- and Computation-Efficient Solver for Lasso Model Fitting with Big Data in R

01/20/2017
by   Yaohui Zeng, et al.
0

Penalized regression models such as the lasso have been extensively applied to analyzing high-dimensional data sets. However, due to memory limitations, existing R packages like glmnet and ncvreg are not capable of fitting lasso-type models for ultrahigh-dimensional, multi-gigabyte data sets that are increasingly seen in many areas such as genetics, genomics, biomedical imaging, and high-frequency finance. In this research, we implement an R package called biglasso that tackles this challenge. biglasso utilizes memory-mapped files to store the massive data on the disk, only reading data into memory when necessary during model fitting, and is thus able to handle out-of-core computation seamlessly. Moreover, it's equipped with newly proposed, more efficient feature screening rules, which substantially accelerate the computation. Benchmarking experiments show that our biglasso package, as compared to existing popular ones like glmnet, is much more memory- and computation-efficient. We further analyze a 31 GB real data set on a laptop with only 16 GB RAM to demonstrate the out-of-core computation capability of biglasso in analyzing massive data sets that cannot be accommodated by existing R packages.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/27/2017

Efficient Feature Screening for Lasso-Type Problems via Hybrid Safe-Strong Rules

The lasso model has been widely used for model selection in data mining,...
research
01/29/2018

Fast Penalized Regression and Cross Validation for Tall Data with the oem Package

A large body of research has focused on theory and computation for varia...
research
06/26/2020

The huge Package for High-dimensional Undirected Graph Estimation in R

We describe an R package named huge which provides easy-to-use functions...
research
01/10/2018

Strong Sure Screening of Ultra-high Dimensional Categorical Data

Feature screening for ultra high dimensional feature spaces plays a crit...
research
01/09/2013

On the Incommensurability Phenomenon

Suppose that two large, multi-dimensional data sets are each noisy measu...
research
06/27/2020

The flare Package for High Dimensional Linear Regression and Precision Matrix Estimation in R

This paper describes an R package named flare, which implements a family...
research
08/16/2019

Visualization of Very Large High-Dimensional Data Sets as Minimum Spanning Trees

Here, we introduce a new data visualization and exploration method, TMAP...

Please sign up or login with your details

Forgot password? Click here to reset