Estimating Stochastic Linear Combination of Non-linear Regressions Efficiently and Scalably

10/19/2020
by   Di Wang, et al.
0

Recently, many machine learning and statistical models such as non-linear regressions, the Single Index, Multi-index, Varying Coefficient Index Models and Two-layer Neural Networks can be reduced to or be seen as a special case of a new model which is called the Stochastic Linear Combination of Non-linear Regressions model. However, due to the high non-convexity of the problem, there is no previous work study how to estimate the model. In this paper, we provide the first study on how to estimate the model efficiently and scalably. Specifically, we first show that with some mild assumptions, if the variate vector x is multivariate Gaussian, then there is an algorithm whose output vectors have ℓ_2-norm estimation errors of O(√(p/n)) with high probability, where p is the dimension of x and n is the number of samples. The key idea of the proof is based on an observation motived by the Stein's lemma. Then we extend our result to the case where x is bounded and sub-Gaussian using the zero-bias transformation, which could be seen as a generalization of the classic Stein's lemma. We also show that with some additional assumptions there is an algorithm whose output vectors have ℓ_∞-norm estimation errors of O(1/√(p)+√(p/n)) with high probability. We also provide a concrete example to show that there exists some link function which satisfies the previous assumptions. Finally, for both Gaussian and sub-Gaussian cases we propose a faster sub-sampling based algorithm and show that when the sub-sample sizes are large enough then the estimation errors will not be sacrificed by too much. Experiments for both cases support our theoretical results. To the best of our knowledge, this is the first work that studies and provides theoretical guarantees for the stochastic linear combination of non-linear regressions model.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/01/2019

Estimating Smooth GLM in Non-interactive Local Differential Privacy Model with Public Unlabeled Data

In this paper, we study the problem of estimating smooth Generalized Lin...
research
10/29/2021

Support Recovery with Stochastic Gates: Theory and Application for Linear Models

We analyze the problem of simultaneous support recovery and estimation o...
research
04/30/2013

Dictionary LASSO: Guaranteed Sparse Recovery under Linear Transformation

We consider the following signal recovery problem: given a measurement m...
research
09/04/2018

High-dimensional varying index coefficient quantile regression model

Statistical learning evolves quickly with more and more sophisticated mo...
research
06/03/2020

Gaussian linear approximation for the estimation of the Shapley effects

In this paper, we address the estimation of the sensitivity indices call...
research
02/24/2019

Nonlinear generalization of the single index model

Single index model is a powerful yet simple model, widely used in statis...
research
01/10/2023

Matching calipers and the precision of index estimation

This paper characterizes the precision of index estimation as it carries...

Please sign up or login with your details

Forgot password? Click here to reset