Multi-Symmetry Ensembles: Improving Diversity and Generalization via Opposing Symmetries

03/04/2023
by   Charlotte Loh, et al.
0

Deep ensembles (DE) have been successful in improving model performance by learning diverse members via the stochasticity of random initialization. While recent works have attempted to promote further diversity in DE via hyperparameters or regularizing loss functions, these methods primarily still rely on a stochastic approach to explore the hypothesis space. In this work, we present Multi-Symmetry Ensembles (MSE), a framework for constructing diverse ensembles by capturing the multiplicity of hypotheses along symmetry axes, which explore the hypothesis space beyond stochastic perturbations of model weights and hyperparameters. We leverage recent advances in contrastive representation learning to create models that separately capture opposing hypotheses of invariant and equivariant symmetries and present a simple ensembling approach to efficiently combine appropriate hypotheses for a given task. We show that MSE effectively captures the multiplicity of conflicting hypotheses that is often required in large, diverse datasets like ImageNet. As a result of their inherent diversity, MSE improves classification performance, uncertainty quantification, and generalization across a series of transfer tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/24/2020

Hyperparameter Ensembles for Robustness and Uncertainty Quantification

Ensembles over neural network weights trained from different random init...
research
12/05/2019

Deep Ensembles: A Loss Landscape Perspective

Deep ensembles have been empirically shown to be a promising approach fo...
research
06/20/2021

On Stein Variational Neural Network Ensembles

Ensembles of deep neural networks have achieved great success recently, ...
research
02/12/2023

Interpretable Diversity Analysis: Visualizing Feature Representations In Low-Cost Ensembles

Diversity is an important consideration in the construction of robust ne...
research
04/08/2023

Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification

We consider the problem of uncertainty quantification in high dimensiona...
research
04/06/2023

Source-free Domain Adaptation Requires Penalized Diversity

While neural networks are capable of achieving human-like performance in...

Please sign up or login with your details

Forgot password? Click here to reset