Efficient Regularized Piecewise-Linear Regression Trees

06/29/2019
by   Leonidas Lefakis, et al.
0

We present a detailed analysis of the class of regression decision tree algorithms which employ a regulized piecewise-linear node-splitting criterion and have regularized linear models at the leaves. From a theoretic standpoint, based on Rademacher complexity framework, we present new high-probability upper bounds for the generalization error for the proposed classes of regularized regression decision tree algorithms, including LASSO-type, and ℓ_2 regularization for linear models at the leaves. Theoretical result are further extended by considering a general type of variable selection procedure. Furthermore, in our work we demonstrate that the class of piecewise-linear regression trees is not only numerically stable but can be made tractable via an algorithmic implementation, presented herein, as well as with the help of modern GPU technology. Empirically, we present results on multiple datasets which highlight the strengths and potential pitfalls, of the proposed tree algorithms compared to baselines which grow trees based on piecewise constant models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/25/2013

A Comprehensive Approach to Universal Piecewise Nonlinear Regression Based on Trees

In this paper, we investigate adaptive nonlinear regression and introduc...
research
05/20/2017

Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression

For various applications, the relations between the dependent and indepe...
research
08/16/2007

Piecewise linear regularized solution paths

We consider the generic regularized optimization problem β̂(λ)=_βL(y,Xβ)...
research
11/16/2020

A family of smooth piecewise-linear models with probabilistic interpretations

The smooth piecewise-linear models cover a wide range of applications no...
research
04/04/2021

Efficient Experimental Design for Regularized Linear Models

Regularized linear models, such as Lasso, have attracted great attention...
research
10/10/2017

LinXGBoost: Extension of XGBoost to Generalized Local Linear Models

XGBoost is often presented as the algorithm that wins every ML competiti...
research
01/20/2021

Autocart – spatially-aware regression trees for ecological and spatial modeling

Many ecological and spatial processes are complex in nature and are not ...

Please sign up or login with your details

Forgot password? Click here to reset