Gradient tree boosting with random output projections for multi-label classification and multi-output regression

05/18/2019
by   Arnaud Joly, et al.
0

In many applications of supervised learning, multiple classification or regression outputs have to be predicted jointly. We consider several extensions of gradient boosting to address such problems. We first propose a straightforward adaptation of gradient boosting exploiting multiple output regression trees as base learners. We then argue that this method is only expected to be optimal when the outputs are fully correlated, as it forces the partitioning induced by the tree base learners to be shared by all outputs. We then propose a novel extension of gradient tree boosting to specifically address this issue. At each iteration of this new method, a regression tree structure is grown to fit a single random projection of the current residuals and the predictions of this tree are fitted linearly to the current residuals of all the outputs, independently. Because of this linear fit, the method can adapt automatically to any output correlation structure. Extensive experiments are conducted with this method, as well as other algorithmic variants, on several artificial and real problems. Randomly projecting the output space is shown to provide a better adaptation to different output correlation patterns and is therefore competitive with the best of the other methods in most settings. Thanks to model sharing, the convergence speed is also improved, reducing the computing times (or the complexity of the model) to reach a specific accuracy.

READ FULL TEXT
research
11/26/2022

Condensed Gradient Boosting

This paper presents a computationally efficient variant of gradient boos...
research
04/14/2014

Random forests with random projections of the output space for high dimensional multi-label classification

We adapt the idea of random projections applied to the output space, so ...
research
09/07/2021

Tree-based boosting with functional data

In this article we propose a boosting algorithm for regression with func...
research
02/11/2019

KTBoost: Combined Kernel and Tree Boosting

In this article, we introduce a novel boosting algorithm called `KTBoost...
research
06/07/2020

Soft Gradient Boosting Machine

Gradient Boosting Machine has proven to be one successful function appro...
research
02/16/2021

Trees-Based Models for Correlated Data

This paper presents a new approach for trees-based regression, such as s...
research
09/10/2019

GBDT-MO: Gradient Boosted Decision Trees for Multiple Outputs

Gradient boosted decision trees (GBDTs) are widely used in machine learn...

Please sign up or login with your details

Forgot password? Click here to reset