Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification

11/21/2017
by   Jean Feng, et al.
0

Neural networks are usually not the tool of choice for nonparametric high-dimensional problems where the number of input features is much larger than the number of observations. Though neural networks can approximate complex multivariate functions, they generally require a large number of training observations to obtain reasonable fits, unless one can learn the appropriate network structure. In this manuscript, we show that neural networks can be applied successfully to high-dimensional settings if the true function falls in a low dimensional subspace, and proper regularization is used. We propose fitting a neural network with a sparse group lasso penalty on the first-layer input weights, which results in a neural net that only uses a small subset of the original features. In addition, we characterize the statistical convergence of the penalized empirical risk minimizer to the optimal neural network: we show that the excess risk of this penalized estimator only grows with the logarithm of the number of input features; and we show that the weights of irrelevant features converge to zero. Via simulation studies and data analyses, we show that these sparse-input neural networks outperform existing nonparametric high-dimensional estimation methods when the data has complex higher-order interactions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/01/2023

Sparse-Input Neural Network using Group Concave Regularization

Simultaneous feature selection and non-linear function estimation are ch...
research
05/11/2020

Ensembled sparse-input hierarchical networks for high-dimensional datasets

Neural networks have seen limited use in prediction for high-dimensional...
research
03/02/2023

Sparse-penalized deep neural networks estimator under weak dependence

We consider the nonparametric regression and the classification problems...
research
12/20/2018

Calibrating Lévy Process from Observations Based on Neural Networks and Automatic Differentiation with Convergence Proofs

The Lévy process has been widely applied to mathematical finance, quantu...
research
08/24/2021

Adaptive Group Lasso Neural Network Models for Functions of Few Variables and Time-Dependent Data

In this paper, we propose an adaptive group Lasso deep neural network fo...
research
07/02/2016

Group Sparse Regularization for Deep Neural Networks

In this paper, we consider the joint task of simultaneously optimizing (...
research
01/06/2018

Generating Neural Networks with Neural Networks

Hypernetworks are neural networks that transform a random input vector i...

Please sign up or login with your details

Forgot password? Click here to reset