A Precise High-Dimensional Asymptotic Theory for Boosting and Min-L1-Norm Interpolated Classifiers

02/05/2020
by   Tengyuan Liang, et al.
0

This paper establishes a precise high-dimensional asymptotic theory for Boosting on separable data, taking statistical and computational perspectives. We consider the setting where the number of features (weak learners) p scales with the sample size n, in an over-parametrized regime. On the statistical front, we provide an exact analysis of the generalization error of Boosting, when the algorithm interpolates the training data and maximizes an empirical L1 margin. The angle between the Boosting solution and the ground truth is characterized explicitly. On the computational front, we provide a sharp analysis of the stopping time when Boosting approximately maximizes the empirical L1 margin. Furthermore, we discover that, the larger the margin, the smaller the proportion of active features (with zero initialization). At the heart of our theory lies a detailed study of the maximum L1 margin, using tools from convex geometry. The maximum L1 margin can be precisely described by a new system of non-linear equations, which we study using a novel uniform deviation argument. Preliminary numerical results are presented to demonstrate the accuracy of our theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/14/2009

Boosting through Optimization of Margin Distributions

Boosting has attracted much research attention in the past decade. The s...
research
03/28/2008

Analysis of boosting algorithms using the smooth margin function

We introduce a useful tool for analyzing boosting algorithms called the ...
research
12/13/2018

On the Differences between L2-Boosting and the Lasso

We prove that L2-Boosting lacks a theoretical property which is central ...
research
12/07/2022

Tight bounds for maximum ℓ_1-margin classifiers

Popular iterative algorithms such as boosting methods and coordinate des...
research
07/04/2013

AdaBoost and Forward Stagewise Regression are First-Order Convex Optimization Methods

Boosting methods are highly popular and effective supervised learning me...
research
01/23/2009

On the Dual Formulation of Boosting Algorithms

We study boosting algorithms from a new perspective. We show that the La...

Please sign up or login with your details

Forgot password? Click here to reset