Fully-Corrective Gradient Boosting with Squared Hinge: Fast Learning Rates and Early Stopping

by   Jinshan Zeng, et al.

Boosting is a well-known method for improving the accuracy of weak learners in machine learning. However, its theoretical generalization guarantee is missing in literature. In this paper, we propose an efficient boosting method with theoretical generalization guarantees for binary classification. Three key ingredients of the proposed boosting method are: a) the fully-corrective greedy (FCG) update in the boosting procedure, b) a differentiable squared hinge (also called truncated quadratic) function as the loss function, and c) an efficient alternating direction method of multipliers (ADMM) algorithm for the associated FCG optimization. The used squared hinge loss not only inherits the robustness of the well-known hinge loss for classification with outliers, but also brings some benefits for computational implementation and theoretical justification. Under some sparseness assumption, we derive a fast learning rate of the order O((m/log m)^-1/4) for the proposed boosting method, which can be further improved to O((m/log m)^-1/2) if certain additional noise assumption is imposed, where m is the size of sample set. Both derived learning rates are the best ones among the existing generalization results of boosting-type methods for classification. Moreover, an efficient early stopping scheme is provided for the proposed method. A series of toy simulations and real data experiments are conducted to verify the developed theories and demonstrate the effectiveness of the proposed method.


page 2

page 3

page 4

page 5

page 6

page 7

page 12

page 13


SGLB: Stochastic Gradient Langevin Boosting

In this paper, we introduce Stochastic Gradient Langevin Boosting (SGLB)...

Fast Training of Effective Multi-class Boosting Using Coordinate Descent Optimization

Wepresentanovelcolumngenerationbasedboostingmethod for multi-class class...

Fast Polynomial Kernel Classification for Massive Data

In the era of big data, it is highly desired to develop efficient machin...

Online Multiclass Boosting

Recent work has extended the theoretical analysis of boosting algorithms...

SPLBoost: An Improved Robust Boosting Algorithm Based on Self-paced Learning

It is known that Boosting can be interpreted as a gradient descent techn...

Kernel-based L_2-Boosting with Structure Constraints

Developing efficient kernel methods for regression is very popular in th...

Randomized Gradient Boosting Machine

Gradient Boosting Machine (GBM) introduced by Friedman is an extremely p...

Please sign up or login with your details

Forgot password? Click here to reset