Learning Near-optimal Convex Combinations of Basis Models with Generalization Guarantees

10/09/2019
by   Tan Nguyen, et al.
0

The problem of learning an optimal convex combination of basis models has been studied in a number of works, with a focus on the theoretical analysis, but little investigation on the empirical performance of the approach. In this paper, we present some new theoretical insights, and empirical results that demonstrate the effectiveness of the approach. Theoretically, we first consider whether we can replace convex combinations by linear combinations, and obtain convergence results similar to existing results for learning from a convex hull. We present a negative result showing that the linear hull of very simple basis functions can have unbounded capacity, and is thus prone to overfitting. On the other hand, convex hulls are still rich but have bounded capacities. In addition, we obtain a generalization bound for a general class of Lipschitz loss functions. Empirically, we first discuss how a convex combination can be greedily learned with early stopping, and how a convex combination can be non-greedily learned when the number of basis models is known a priori. Our experiments suggest that the greedy scheme is competitive with or better than several baselines, including boosting and random forests. The greedy algorithm requires little effort in hyper-parameter tuning, and also seems to adapt to the underlying complexity of the problem.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/05/2015

Boosting in the presence of outliers: adaptive classification with non-convex loss functions

This paper examines the role and efficiency of the non-convex loss funct...
research
01/30/2022

Faster Convergence of Local SGD for Over-Parameterized Models

Modern machine learning architectures are often highly expressive. They ...
research
03/31/2015

Iterative Regularization for Learning with Convex Loss Functions

We consider the problem of supervised learning with convex loss function...
research
05/31/2017

Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees

Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolf...
research
02/14/2012

Ensembles of Kernel Predictors

This paper examines the problem of learning with a finite and possibly l...
research
02/17/2020

Sharp Asymptotics and Optimal Performance for Inference in Binary Models

We study convex empirical risk minimization for high-dimensional inferen...
research
05/09/2018

On the Construction of Substitutes

Gross substitutability is a central concept in Economics and is connecte...

Please sign up or login with your details

Forgot password? Click here to reset