All of Linear Regression

by   Arun K. Kuchibhotla, et al.

Least squares linear regression is one of the oldest and widely used data analysis tools. Although the theoretical analysis of the ordinary least squares (OLS) estimator is as old, several fundamental questions are yet to be answered. Suppose regression observations (X_1,Y_1),...,(X_n,Y_n)∈R^d×R (not necessarily independent) are available. Some of the questions we deal with are as follows: under what conditions, does the OLS estimator converge and what is the limit? What happens if the dimension is allowed to grow with n? What happens if the observations are dependent with dependence possibly strengthening with n? How to do statistical inference under these kinds of misspecification? What happens to the OLS estimator under variable selection? How to do inference under misspecification and variable selection? We answer all the questions raised above with one simple deterministic inequality which holds for any set of observations and any sample size. This implies that all our results are a finite sample (non-asymptotic) in nature. In the end, one only needs to bound certain random quantities under specific settings of interest to get concrete rates and we derive these bounds for the case of independent observations. In particular, the problem of inference after variable selection is studied, for the first time, when d, the number of covariates increases (almost exponentially) with sample size n. We provide comments on the “right” statistic to consider for inference under variable selection and efficient computation of quantiles.


page 1

page 2

page 3

page 4


Inference for Projection Parameters in Linear Regression: beyond d = o(n^1/2)

We consider the problem of inference for projection parameters in linear...

A Model Free Perspective for Linear Regression: Uniform-in-model Bounds for Post Selection Inference

For the last two decades, high-dimensional data and methods have prolife...

A Power Analysis of the Conditional Randomization Test and Knockoffs

In many scientific problems, researchers try to relate a response variab...

Nonparametric Assessment of Variable Selection and Ranking Algorithms

Selecting from or ranking a set of candidates variables in terms of thei...

Forbidden Knowledge and Specialized Training: A Versatile Solution for the Two Main Sources of Overfitting in Linear Regression

Overfitting in linear regression is broken down into two main causes. Fi...

Poisson Subsampling Algorithms for Large Sample Linear Regression in Massive Data

Large sample size brings the computation bottleneck for modern data anal...

A Model-free Approach to Linear Least Squares Regression with Exact Probabilities and Applications to Covariate Selection

The classical model for linear regression is Y= xβ +σε with i.i.d. stan...

Please sign up or login with your details

Forgot password? Click here to reset