Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordance

02/08/2019
by   Ulysse Marteau-Ferey, et al.
0

We consider learning methods based on the regularization of a convex empirical risk by a squared Hilbertian norm, a setting that includes linear predictors and non-linear predictors through positive-definite kernels. In order to go beyond the generic analysis leading to convergence rates of the excess risk as O(1/√(n)) from n observations, we assume that the individual losses are self-concordant, that is, their third-order derivatives are bounded by their second-order derivatives. This setting includes least-squares, as well as all generalized linear models such as logistic and softmax regression. For this class of losses, we provide a bias-variance decomposition and show that the assumptions commonly made in least-squares regression, such as the source and capacity conditions, can be adapted to obtain fast non-asymptotic rates of convergence by improving the bias terms, the variance terms or both.

READ FULL TEXT
research
06/16/2021

Beyond Tikhonov: Faster Learning with Self-Concordant Losses via Iterative Regularization

The theory of spectral filtering is a remarkable tool to understand the ...
research
07/03/2019

Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses

In this paper, we study large-scale convex optimization algorithms based...
research
02/23/2017

Sobolev Norm Learning Rates for Regularized Least-Squares Algorithm

Learning rates for regularized least-squares algorithms are in most case...
research
09/19/2020

Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors

We study the problem of predicting as well as the best linear predictor ...
research
05/29/2023

On the Variance, Admissibility, and Stability of Empirical Risk Minimization

It is well known that Empirical Risk Minimization (ERM) with squared los...
research
02/03/2022

Multiclass learning with margin: exponential rates with no bias-variance trade-off

We study the behavior of error bounds for multiclass classification unde...
research
05/10/2016

Learning theory estimates with observations from general stationary stochastic processes

This paper investigates the supervised learning problem with observation...

Please sign up or login with your details

Forgot password? Click here to reset