MABSplit: Faster Forest Training Using Multi-Armed Bandits

12/14/2022
by   Mo Tiwari, et al.
0

Random forests are some of the most widely used machine learning models today, especially in domains that necessitate interpretability. We present an algorithm that accelerates the training of random forests and other popular tree-based learning methods. At the core of our algorithm is a novel node-splitting subroutine, dubbed MABSplit, used to efficiently find split points when constructing decision trees. Our algorithm borrows techniques from the multi-armed bandit literature to judiciously determine how to allocate samples and computational power across candidate split points. We provide theoretical guarantees that MABSplit improves the sample complexity of each node split from linear to logarithmic in the number of data points. In some settings, MABSplit leads to 100x faster training (an 99 time) without any decrease in generalization performance. We demonstrate similar speedups when MABSplit is used across a variety of forest-based variants, such as Extremely Random Forests and Random Patches. We also show our algorithm can be used in both classification and regression tasks. Finally, we show that MABSplit outperforms existing methods in generalization performance and feature importance calculations under a fixed computational budget. All of our experimental results are reproducible via a one-line script at https://github.com/ThrunGroup/FastForest.

READ FULL TEXT

page 22

page 23

page 24

research
03/10/2019

Multinomial Random Forests: Fill the Gap between Theoretical Consistency and Empirical Soundness

Random forests (RF) are one of the most widely used ensemble learning me...
research
06/19/2015

CO2 Forest: Improved Random Forest by Continuous Optimization of Oblique Splits

We propose a novel algorithm for optimizing multivariate linear threshol...
research
05/12/2015

The Boundary Forest Algorithm for Online Supervised and Unsupervised Learning

We describe a new instance-based learning algorithm called the Boundary ...
research
11/02/2017

Medoids in almost linear time via multi-armed bandits

Computing the medoid of a large number of points in high-dimensional spa...
research
12/19/2013

Learning Transformations for Classification Forests

This work introduces a transformation-based learner model for classifica...
research
09/30/2020

Uncovering Feature Interdependencies in Complex Systems with Non-Greedy Random Forests

A "non-greedy" variation of the random forest algorithm is presented to ...
research
07/04/2023

MDI+: A Flexible Random Forest-Based Feature Importance Framework

Mean decrease in impurity (MDI) is a popular feature importance measure ...

Please sign up or login with your details

Forgot password? Click here to reset