A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning

08/05/2019
by   Lesia Semenova, et al.
0

The Rashomon effect occurs when many different explanations exist for the same phenomenon. In machine learning, Leo Breiman used this term to describe problems where many accurate-but-different models exist to describe the same data. In this work, we study how the Rashomon effect can be useful for understanding the relationship between training and test performance, and the possibility that simple-yet-accurate models exist for many problems. We introduce the Rashomon set as the set of almost-equally-accurate models for a given problem, and study its properties and the types of models it could contain. We present the Rashomon ratio as a new measure related to simplicity of model classes, which is the ratio of the volume of the set of accurate models to the volume of the hypothesis space; the Rashomon ratio is different from standard complexity measures from statistical learning theory. For a hierarchy of hypothesis spaces, the Rashomon ratio can help modelers to navigate the trade-off between simplicity and accuracy in a surprising way. In particular, we find empirically that a plot of empirical risk vs. Rashomon ratio forms a characteristic Γ-shaped Rashomon curve, whose elbow seems to be a reliable model selection criterion. When the Rashomon set is large, models that are accurate - but that also have various other useful properties - can often be obtained. These models might obey various constraints such as interpretability, fairness, monotonicity, and computational benefits.

READ FULL TEXT
06/08/2020

A Geometric Look at Double Descent Risk: Volumes, Singularities, and Distinguishabilities

The appearance of the double-descent risk phenomenon has received growin...
04/11/2020

Grounding Occam's Razor in a Formal Theory of Simplicity

It is proposed that the Occam's Razor heuristic – when in doubt, choose ...
04/01/2019

Fast, accurate, and transferable many-body interatomic potentials by genetic programming

The length and time scales of atomistic simulations are limited by the c...
04/01/2019

Fast, accurate, and transferable many-body interatomic potentials by symbolic regression

The length and time scales of atomistic simulations are limited by the c...
05/05/2019

Decision Making with Machine Learning and ROC Curves

The Receiver Operating Characteristic (ROC) curve is a representation of...
07/01/2022

Characterizing the Effect of Class Imbalance on the Learning Dynamics

Data imbalance is a common problem in the machine learning literature th...
06/22/2018

Learning Qualitatively Diverse and Interpretable Rules for Classification

There has been growing interest in developing accurate models that can a...