DeepAI
Log In Sign Up

Classification vs regression in overparameterized regimes: Does the loss function matter?

05/16/2020
by   Vidya Muthukumar, et al.
40

We compare classification and regression tasks in the overparameterized linear model with Gaussian features. On the one hand, we show that with sufficient overparameterization all training points are support vectors: solutions obtained by least-squares minimum-norm interpolation, typically used for regression, are identical to those produced by the hard-margin support vector machine (SVM) that minimizes the hinge loss, typically used for training classifiers. On the other hand, we show that there exist regimes where these solutions are near-optimal when evaluated by the 0-1 test loss function, but do not generalize if evaluated by the square loss function, i.e. they achieve the null risk. Our results demonstrate the very different roles and properties of loss functions used at the training phase (optimization) and the testing phase (generalization).

READ FULL TEXT

page 1

page 2

page 3

page 4

04/05/2019

Logitron: Perceptron-augmented classification model based on an extended logistic loss function

Classification is the most important process in data analysis. However, ...
06/21/2021

Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation

The growing literature on "benign overfitting" in overparameterized mode...
04/28/2019

Support Vector Regression via a Combined Reward Cum Penalty Loss Function

In this paper, we introduce a novel combined reward cum penalty loss fun...
04/30/2012

A Conjugate Property between Loss Functions and Uncertainty Sets in Classification Problems

In binary classification problems, mainly two approaches have been propo...
07/15/2022

Support Vector Machines with the Hard-Margin Loss: Optimal Training via Combinatorial Benders' Cuts

The classical hinge-loss support vector machines (SVMs) model is sensiti...
05/24/2022

Soft-SVM Regression For Binary Classification

The binomial deviance and the SVM hinge loss functions are two of the mo...
01/29/2019

Hyperspherical Prototype Networks

This paper introduces hyperspherical prototype networks, which unify reg...