DeepAI AI Chat
Log In Sign Up

Dimensionality reduction, regularization, and generalization in overparameterized regressions

11/23/2020
by   Ningyuan Huang, et al.
Johns Hopkins University
0

Overparameterization in deep learning is powerful: Very large models fit the training data perfectly and yet generalize well. This realization brought back the study of linear models for regression, including ordinary least squares (OLS), which, like deep learning, shows a "double descent" behavior. This involves two features: (1) The risk (out-of-sample prediction error) can grow arbitrarily when the number of samples n approaches the number of parameters p, and (2) the risk decreases with p at p>n, sometimes achieving a lower value than the lowest risk at p<n. The divergence of the risk for OLS at p≈ n is related to the condition number of the empirical covariance in the feature set. For this reason, it can be avoided with regularization. In this work we show that it can also be avoided with a PCA-based dimensionality reduction. We provide a finite upper bound for the risk of the PCA-based estimator. This result is in contrast to recent work that shows that a different form of dimensionality reduction – one based on the population covariance instead of the empirical covariance – does not avoid the divergence. We connect these results to an analysis of adversarial attacks, which become more effective as they raise the condition number of the empirical covariance of the features. We show that OLS is arbitrarily susceptible to data-poisoning attacks in the overparameterized regime – unlike the underparameterized regime – and that regularization and dimensionality reduction improve the robustness.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/18/2020

The Dilemma Between Dimensionality Reduction and Adversarial Robustness

Recent work has shown the tremendous vulnerability to adversarial sample...
09/29/2015

Foundations of Coupled Nonlinear Dimensionality Reduction

In this paper we introduce and analyze the learning scenario of coupled ...
12/19/2022

Prediction of Auto Insurance Risk Based on t-SNE Dimensionality Reduction

Correct scoring of a driver's risk is of great significance to auto insu...
02/02/2020

Overfitting Can Be Harmless for Basis Pursuit: Only to a Degree

Recently, there have been significant interests in studying the generali...
02/12/2013

Adaptive Metric Dimensionality Reduction

We study adaptive data-dependent dimensionality reduction in the context...
10/31/2018

The Price of Fair PCA: One Extra Dimension

We investigate whether the standard dimensionality reduction technique o...
03/04/2020

Visualizing and Understanding Large-Scale Assessments in Mathematics through Dimensionality Reduction

In this paper, we apply the Logistic PCA (LPCA) as a dimensionality redu...