Diversity and Generalization in Neural Network Ensembles

by   Luis A. Ortega, et al.

Ensembles are widely used in machine learning and, usually, provide state-of-the-art performance in many prediction tasks. From the very beginning, the diversity of an ensemble has been identified as a key factor for the superior performance of these models. But the exact role that diversity plays in ensemble models is poorly understood, specially in the context of neural networks. In this work, we combine and expand previously published results in a theoretically sound framework that describes the relationship between diversity and ensemble performance for a wide range of ensemble methods. More precisely, we provide sound answers to the following questions: how to measure diversity, how diversity relates to the generalization error of an ensemble, and how diversity is promoted by neural network ensemble algorithms. This analysis covers three widely used loss functions, namely, the squared loss, the cross-entropy loss, and the 0-1 loss; and two widely used model combination strategies, namely, model averaging and weighted majority vote. We empirically validate this theoretical analysis with neural network ensembles.


page 1

page 2

page 3

page 4


When does Diversity Help Generalization in Classification Ensembles?

Ensembles, as a widely used and effective technique in the machine learn...

EnFuzz: From Ensemble Learning to Ensemble Fuzzing

Fuzzing is widely used for software vulnerability detection. There are v...

Cascade Neural Ensemble for Identifying Scientifically Sound Articles

Background: A significant barrier to conducting systematic reviews and m...

The Ghost of Performance Reproducibility Past

The importance of ensemble computing is well established. However, execu...

Diversity of Ensembles for Data Stream Classification

When constructing a classifier ensemble, diversity among the base classi...

A Unified Theory of Diversity in Ensemble Learning

We present a theory of ensemble diversity, explaining the nature and eff...

Diversity and degrees of freedom in regression ensembles

Ensemble methods are a cornerstone of modern machine learning. The perfo...

Please sign up or login with your details

Forgot password? Click here to reset