The Geometry and Calculus of Losses

09/01/2022
by   Robert C. Williamson, et al.
20

Statistical decision problems are the foundation of statistical machine learning. The simplest problems are binary and multiclass classification and class probability estimation. Central to their definition is the choice of loss function, which is the means by which the quality of a solution is evaluated. In this paper we systematically develop the theory of loss functions for such problems from a novel perspective whose basic ingredients are convex sets with a particular structure. The loss function is defined as the subgradient of the support function of the convex set. It is consequently automatically proper (calibrated for probability estimation). This perspective provides three novel opportunities. It enables the development of a fundamental relationship between losses and (anti)-norms that appears to have not been noticed before. Second, it enables the development of a calculus of losses induced by the calculus of convex sets which allows the interpolation between different losses, and thus is a potential useful design tool for tailoring losses to particular problems. In doing this we build upon, and considerably extend, existing results on M-sums of convex sets. Third, the perspective leads to a natural theory of `polar' (or `inverse') loss functions, which are derived from the polar dual of the convex set defining the loss, and which form a natural universal substitution function for Vovk's aggregating algorithm.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/27/2023

LegendreTron: Uprising Proper Multiclass Loss Learning

Loss functions serve as the foundation of supervised learning and are of...
research
12/24/2015

The Lovász Hinge: A Novel Convex Surrogate for Submodular Losses

Learning with non-modular losses is an important problem when sets of pr...
research
12/17/2009

Composite Binary Losses

We study losses for binary classification and class probability estimati...
research
10/24/2019

Structured Prediction with Projection Oracles

We propose in this paper a general framework for deriving loss functions...
research
06/18/2021

Being Properly Improper

In today's ML, data can be twisted (changed) in various ways, either for...
research
02/10/2020

Supervised Learning: No Loss No Cry

Supervised learning requires the specification of a loss function to min...
research
03/13/2023

General Loss Functions Lead to (Approximate) Interpolation in High Dimensions

We provide a unified framework, applicable to a general family of convex...

Please sign up or login with your details

Forgot password? Click here to reset