DeepAI

Behavior of linear L2-boosting algorithms in the vanishing learning rate asymptotic

We investigate the asymptotic behaviour of gradient boosting algorithms when the learning rate converges to zero and the number of iterations is rescaled accordingly. We mostly consider L2-boosting for regression with linear base learner as studied in Bühlmann and Yu (2003) and analyze also a stochastic version of the model where subsampling is used at each step (Friedman 2002). We prove a deterministic limit in the vanishing learning rate asymptotic and characterize the limit as the unique solution of a linear differential equation in an infinite dimensional function space. Besides, the training and test error of the limiting procedure are thoroughly analyzed. We finally illustrate and discuss our result on a simple numerical experiment where the linear L2-boosting operator is interpreted as a smoothed projection and time is related to its number of degrees of freedom.

• 11 publications
• 5 publications
10/03/2022

A large sample theory for infinitesimal gradient boosting

Infinitesimal gradient boosting is defined as the vanishing-learning-rat...
04/26/2021

We define infinitesimal gradient boosting as a limit of the popular tree...
04/15/2020

On Learning Rates and Schrödinger Operators

The learning rate is perhaps the single most important parameter in the ...
05/16/2015

A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives

In this paper we analyze boosting algorithms in linear regression from a...
07/19/2017

Acceleration and Averaging in Stochastic Mirror Descent Dynamics

We formulate and study a general family of (continuous-time) stochastic ...
11/06/2019

Improving reinforcement learning algorithms: towards optimal learning rate policies

This paper investigates to what extent we can improve reinforcement lear...
09/26/2019

Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samples

Gradient-based temporal difference (GTD) algorithms are widely used in o...