Rethinking Fano's Inequality in Ensemble Learning

05/25/2022
by   Terufumi Morishita, et al.
0

We propose a fundamental theory on ensemble learning that evaluates a given ensemble system by a well-grounded set of metrics. Previous studies used a variant of Fano's inequality of information theory and derived a lower bound of the classification error rate on the basis of the accuracy and diversity of models. We revisit the original Fano's inequality and argue that the studies did not take into account the information lost when multiple model predictions are combined into a final prediction. To address this issue, we generalize the previous theory to incorporate the information loss. Further, we empirically validate and demonstrate the proposed theory through extensive experiments on actual systems. The theory reveals the strengths and weaknesses of systems on each metric, which will push the theoretical understanding of ensemble learning and give us insights into designing systems.

READ FULL TEXT
research
06/19/2023

AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing

In this paper, we examine the current state-of-the-art in AMR parsing, w...
research
02/11/2004

A Numerical Example on the Principles of Stochastic Discrimination

Studies on ensemble methods for classification suffer from the difficult...
research
10/14/2022

On Triangular Inequality of the Discounted Least Information Theory of Entropy (DLITE)

The Discounted Least Information Theory of Entropy (DLITE) is a new info...
research
01/10/2023

A Unified Theory of Diversity in Ensemble Learning

We present a theory of ensemble diversity, explaining the nature and eff...
research
06/09/2021

Loss function based second-order Jensen inequality and its application to particle variational inference

Bayesian model averaging, obtained as the expectation of a likelihood fu...
research
05/26/2019

Ensemble Decision Systems for General Video Game Playing

Ensemble Decision Systems offer a unique form of decision making that al...
research
12/20/2016

WoCE: a framework for clustering ensemble by exploiting the wisdom of Crowds theory

The Wisdom of Crowds (WOC), as a theory in the social science, gets a ne...

Please sign up or login with your details

Forgot password? Click here to reset