DeepAI
Log In Sign Up

Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression

06/03/2021
by   Olivier Sprangers, et al.
24

Gradient Boosting Machines (GBM) are hugely popular for solving tabular data problems. However, practitioners are not only interested in point predictions, but also in probabilistic predictions in order to quantify the uncertainty of the predictions. Creating such probabilistic predictions is difficult with existing GBM-based solutions: they either require training multiple models or they become too computationally expensive to be useful for large-scale settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of decision trees in a computationally efficient manner. PGBM approximates the leaf weights in a decision tree as a random variable, and approximates the mean and variance of each sample in a dataset via stochastic tree ensemble update equations. These learned moments allow us to subsequently sample from a specified distribution after training. We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods: (i) PGBM enables probabilistic estimates without compromising on point performance in a single model, (ii) PGBM learns probabilistic estimates via a single model only (and without requiring multi-parameter boosting), and thereby offers a speedup of up to several orders of magnitude over existing state-of-the-art methods on large datasets, and (iii) PGBM achieves accurate probabilistic estimates in tasks with complex differentiable loss functions, such as hierarchical time series problems, where we observed up to 10 300

READ FULL TEXT

page 1

page 2

page 3

page 4

06/18/2020

Uncertainty in Gradient Boosting via Ensembles

Gradient boosting is a powerful machine learning technique that is parti...
05/23/2022

Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees

We propose Instance-Based Uncertainty estimation for Gradient-boosted re...
04/02/2022

Distributional Gradient Boosting Machines

We present a unified probabilistic gradient boosting framework for regre...
05/02/2020

Large-scale Uncertainty Estimation and Its Application in Revenue Forecast of SMEs

The economic and banking importance of the small and medium enterprise (...
10/26/2020

Versatile Verification of Tree Ensembles

Machine learned models often must abide by certain requirements (e.g., f...
06/07/2021

Multivariate Probabilistic Regression with Natural Gradient Boosting

Many single-target regression problems require estimates of uncertainty ...
10/14/2020

Interpretable Machine Learning with an Ensemble of Gradient Boosting Machines

A method for the local and global interpretation of a black-box model on...

Code Repositories

pgbm

Probabilistic Gradient Boosting Machines


view repo