Gradient Bagging

What is Gradient Bagging?

Gradient bagging, also called Bootstrap Aggregation, is a metaheuristic algorithm that reduces variance and overfitting in a deep learning program. While usually applied to decision trees, bagging can be used in any model. In this approach, several random subsets of data are created from the training sample. Each collection of subset data is then deployed to train a different decision tree. The end result is an ensemble of different models, with the average of all decision tree predictions used instead of just one. Bagging is also used for the node splitting step when creating Random Forests.

What’s the Difference Between Bagging and Gradient Boosting?

Gradient boosting is also an ensemble technique that creates a random collection of predictor models. In this approach though, decision trees are trained sequentially. At each step, the results are analyzed and the model updated for a better “fit” to reduce errors. While boosting is more accurate than bagging, it also leads to more overfitting and variance than bagging.