SGLB: Stochastic Gradient Langevin Boosting

01/20/2020
by   Aleksei Ustimenko, et al.
1

In this paper, we introduce Stochastic Gradient Langevin Boosting (SGLB) - a powerful and efficient machine learning framework, which may deal with a wide range of loss functions and has provable generalization guarantees. The method is based on a special form of Langevin Diffusion equation specifically designed for gradient boosting. This allows us to guarantee the global convergence, while standard gradient boosting algorithms can guarantee only local optima, which is a problem for multimodal loss functions. To illustrate the advantages of SGLB, we apply it to a classification task with 0-1 loss function, which is known to be multimodal, and to a standard Logistic regression task that is convex. The algorithm is implemented as a part of the CatBoost gradient boosting library and outperforms classic gradient boosting methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/04/2020

StochasticRank: Global Optimization of Scale-Free Discrete Functions

In this paper, we introduce a powerful and efficient framework for the d...
research
08/19/2019

Gradient Boosting Machine: A Survey

In this survey, we discuss several different types of gradient boosting ...
research
02/19/2020

Gradient Boosting Neural Networks: GrowNet

A novel gradient boosting framework is proposed where shallow neural net...
research
09/28/2022

TRBoost: A Generic Gradient Boosting Machine based on Trust-region Method

A generic Gradient Boosting Machine called Trust-region Boosting (TRBoos...
research
04/01/2020

Fully-Corrective Gradient Boosting with Squared Hinge: Fast Learning Rates and Early Stopping

Boosting is a well-known method for improving the accuracy of weak learn...
research
05/07/2018

Wavelet Decomposition of Gradient Boosting

In this paper we introduce a significant improvement to the popular tree...
research
06/20/2017

SPLBoost: An Improved Robust Boosting Algorithm Based on Self-paced Learning

It is known that Boosting can be interpreted as a gradient descent techn...

Please sign up or login with your details

Forgot password? Click here to reset