DeepAI AI Chat
Log In Sign Up

Decision tree heuristics can fail, even in the smoothed setting

by   Guy Blanc, et al.

Greedy decision tree learning heuristics are mainstays of machine learning practice, but theoretical justification for their empirical success remains elusive. In fact, it has long been known that there are simple target functions for which they fail badly (Kearns and Mansour, STOC 1996). Recent work of Brutzkus, Daniely, and Malach (COLT 2020) considered the smoothed analysis model as a possible avenue towards resolving this disconnect. Within the smoothed setting and for targets f that are k-juntas, they showed that these heuristics successfully learn f with depth-k decision tree hypotheses. They conjectured that the same guarantee holds more generally for targets that are depth-k decision trees. We provide a counterexample to this conjecture: we construct targets that are depth-k decision trees and show that even in the smoothed setting, these heuristics build trees of depth 2^Ω(k) before achieving high accuracy. We also show that the guarantees of Brutzkus et al. cannot extend to the agnostic setting: there are targets that are very close to k-juntas, for which these heuristics build trees of depth 2^Ω(k) before achieving high accuracy.


page 1

page 2

page 3

page 4


Properly learning decision trees in almost polynomial time

We give an n^O(loglog n)-time membership query algorithm for properly an...

Regularized impurity reduction: Accurate decision trees with complexity guarantees

Decision trees are popular classification models, providing high accurac...

Provable guarantees for decision tree induction: the agnostic setting

We give strengthened provable guarantees on the performance of widely em...

Learning Optimal Decision Trees from Large Datasets

Inferring a decision tree from a given dataset is one of the classic pro...

Estimating decision tree learnability with polylogarithmic sample complexity

We show that top-down decision tree learning heuristics are amenable to ...

Random Forests, Decision Trees, and Categorical Predictors: The "Absent Levels" Problem

One of the advantages that decision trees have over many other models is...

Optimal Resampling for Learning Small Models

Models often need to be constrained to a certain size for them to be con...