Log In Sign Up

Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression

by   Olivier Sprangers, et al.

Gradient Boosting Machines (GBM) are hugely popular for solving tabular data problems. However, practitioners are not only interested in point predictions, but also in probabilistic predictions in order to quantify the uncertainty of the predictions. Creating such probabilistic predictions is difficult with existing GBM-based solutions: they either require training multiple models or they become too computationally expensive to be useful for large-scale settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of decision trees in a computationally efficient manner. PGBM approximates the leaf weights in a decision tree as a random variable, and approximates the mean and variance of each sample in a dataset via stochastic tree ensemble update equations. These learned moments allow us to subsequently sample from a specified distribution after training. We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods: (i) PGBM enables probabilistic estimates without compromising on point performance in a single model, (ii) PGBM learns probabilistic estimates via a single model only (and without requiring multi-parameter boosting), and thereby offers a speedup of up to several orders of magnitude over existing state-of-the-art methods on large datasets, and (iii) PGBM achieves accurate probabilistic estimates in tasks with complex differentiable loss functions, such as hierarchical time series problems, where we observed up to 10 300


page 1

page 2

page 3

page 4


Uncertainty in Gradient Boosting via Ensembles

Gradient boosting is a powerful machine learning technique that is parti...

Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees

We propose Instance-Based Uncertainty estimation for Gradient-boosted re...

Distributional Gradient Boosting Machines

We present a unified probabilistic gradient boosting framework for regre...

Large-scale Uncertainty Estimation and Its Application in Revenue Forecast of SMEs

The economic and banking importance of the small and medium enterprise (...

Versatile Verification of Tree Ensembles

Machine learned models often must abide by certain requirements (e.g., f...

Multivariate Probabilistic Regression with Natural Gradient Boosting

Many single-target regression problems require estimates of uncertainty ...

Interpretable Machine Learning with an Ensemble of Gradient Boosting Machines

A method for the local and global interpretation of a black-box model on...

Code Repositories


Probabilistic Gradient Boosting Machines

view repo