DeepAI AI Chat
Log In Sign Up

Ensembles of Locally Independent Prediction Models

11/04/2019
by   Andrew Slavin Ross, et al.
28

Many ensemble methods encourage their constituent models to be diverse, because ensembling provides no benefits when models are identical. Most methods define diversity in terms of differences in training set predictions. In this paper, however, we demonstrate that diversity in training set predictions does not necessarily imply diversity when extrapolating even slightly outside it (which can affect generalization). To address this issue, we introduce a new diversity metric and associated method of training ensembles of models that extrapolate differently on local patches of the data manifold. Across a variety of synthetic and real-world tasks, we find that our method improves generalization and diversity in qualitatively novel ways, especially under data limits and covariate shift.

READ FULL TEXT

page 3

page 6

page 11

06/15/2020

Neural Ensemble Search for Performant and Calibrated Predictions

Ensembles of neural networks achieve superior performance compared to st...
02/12/2023

Interpretable Diversity Analysis: Visualizing Feature Representations In Low-Cost Ensembles

Diversity is an important consideration in the construction of robust ne...
10/30/2019

When does Diversity Help Generalization in Classification Ensembles?

Ensembles, as a widely used and effective technique in the machine learn...
02/01/2023

Pathologies of Predictive Diversity in Deep Ensembles

Classical results establish that ensembles of small models benefit when ...
02/22/2018

Diversity regularization in deep ensembles

Calibrating the confidence of supervised learning models is important fo...
03/15/2023

Exploring Resiliency to Natural Image Corruptions in Deep Learning using Design Diversity

In this paper, we investigate the relationship between diversity metrics...
10/19/2020

Anti-Distillation: Improving reproducibility of deep networks

Deep networks have been revolutionary in improving performance of machin...

Code Repositories

lit

Code for AAAI 2020 paper "Ensembles of Locally Independent Prediction Models"


view repo