Top-down induction of decision trees: rigorous guarantees and inherent limitations

11/18/2019
by   Guy Blanc, et al.
0

Consider the following heuristic for building a decision tree for a function f : {0,1}^n →{± 1}. Place the most influential variable x_i of f at the root, and recurse on the subfunctions f_x_i=0 and f_x_i=1 on the left and right subtrees respectively; terminate once the tree is an ε-approximation of f. We analyze the quality of this heuristic, obtaining near-matching upper and lower bounds: ∘ Upper bound: For every f with decision tree size s and every ε∈ (0,1/2), this heuristic builds a decision tree of size at most s^O(log(s/ε)log(1/ε)). ∘ Lower bound: For every ε∈ (0,1/2) and s < 2^Õ(√(n)), there is an f with decision tree size s such that this heuristic builds a decision tree of size s^Ω̃(log s). We also obtain upper and lower bounds for monotone functions: s^O(√(log s)/ε) and s^Ω̃(√(log s ) ) respectively. The lower bound disproves conjectures of Fiat and Pechyony (2004) and Lee (2009). Our upper bounds yield new algorithms for properly learning decision trees under the uniform distribution. We show that these algorithms—which are motivated by widely employed and empirically successful top-down decision tree learning heuristics such as ID3, C4.5, and CART—achieve provable guarantees that compare favorably with those of the current fastest algorithm (Ehrenfeucht and Haussler, 1989). Our lower bounds shed new light on the limitations of these heuristics. Finally, we revisit the classic work of Ehrenfeucht and Haussler. We extend it to give the first uniform-distribution proper learning algorithm that achieves polynomial sample and memory complexity, while matching its state-of-the-art quasipolynomial runtime.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/01/2021

Properly learning decision trees in almost polynomial time

We give an n^O(loglog n)-time membership query algorithm for properly an...
research
06/01/2020

Provable guarantees for decision tree induction: the agnostic setting

We give strengthened provable guarantees on the performance of widely em...
research
06/17/2022

Popular decision tree algorithms are provably noise tolerant

Using the framework of boosting, we prove that all impurity-based decisi...
research
01/20/2023

Superpolynomial Lower Bounds for Learning Monotone Classes

Koch, Strassle, and Tan [SODA 2023], show that, under the randomized exp...
research
10/16/2020

Universal guarantees for decision tree induction via a higher-order splitting criterion

We propose a simple extension of top-down decision tree learning heurist...
research
11/09/2020

Solving the Steiner Tree Problem with few Terminals

The Steiner tree problem is a well-known problem in network design, rout...
research
08/17/2023

Simpler Analyses of Union-Find

We analyze union-find using potential functions motivated by continuous ...

Please sign up or login with your details

Forgot password? Click here to reset