Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors

02/25/2020
by   Yehuda Dar, et al.
12

We study the linear subspace fitting problem in the overparameterized setting, where the estimated subspace can perfectly interpolate the training examples. Our scope includes the least-squares solutions to subspace fitting tasks with varying levels of supervision in the training data (i.e., the proportion of input-output examples of the desired low-dimensional mapping) and orthonormality of the vectors defining the learned operator. This flexible family of problems connects standard, unsupervised subspace fitting that enforces strict orthonormality with a corresponding regression task that is fully supervised and does not constrain the linear operator structure. This class of problems is defined over a supervision-orthonormality plane, where each coordinate induces a problem instance with a unique pair of supervision level and softness of orthonormality constraints. We explore this plane and show that the generalization errors of the corresponding subspace fitting problems follow double descent trends as the settings become more supervised and less orthonormally constrained.

READ FULL TEXT

page 1

page 5

page 17

page 18

research
06/07/2021

Double Descent and Other Interpolation Phenomena in GANs

We study overparameterization in generative adversarial networks (GANs) ...
research
06/12/2020

Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks

We study the transfer learning process between two linear regression pro...
research
02/19/2014

Subspace Learning with Partial Information

The goal of subspace learning is to find a k-dimensional subspace of R^d...
research
06/10/2022

Extremal Fitting Problems for Conjunctive Queries

The fitting problem for conjunctive queries (CQs) is the problem to cons...
research
12/30/2011

An Amendment of Fast Subspace Tracking Methods

Tuning stepsize between convergence rate and steady state error level or...
research
09/02/2022

How Descriptive are GMRES Convergence Bounds?

GMRES is a popular Krylov subspace method for solving linear systems of ...

Please sign up or login with your details

Forgot password? Click here to reset