Bounding generalization error with input compression: An empirical study with infinite-width networks

07/19/2022
by   Angus Galloway, et al.
0

Estimating the Generalization Error (GE) of Deep Neural Networks (DNNs) is an important task that often relies on availability of held-out data. The ability to better predict GE based on a single training set may yield overarching DNN design principles to reduce a reliance on trial-and-error, along with other performance assessment advantages. In search of a quantity relevant to GE, we investigate the Mutual Information (MI) between the input and final layer representations, using the infinite-width DNN limit to bound MI. An existing input compression-based GE bound is used to link MI and GE. To the best of our knowledge, this represents the first empirical study of this bound. In our attempt to empirically falsify the theoretical bound, we find that it is often tight for best-performing models. Furthermore, it detects randomization of training labels in many cases, reflects test-time perturbation robustness, and works well given only few training samples. These results are promising given that input compression is broadly applicable where MI can be estimated with confidence.

READ FULL TEXT

page 19

page 21

page 22

page 23

research
10/12/2018

Estimating Information Flow in Neural Networks

We study the flow of information and the evolution of internal represent...
research
06/18/2021

A Probabilistic Representation of DNNs: Bridging Mutual Information and Generalization

Recently, Mutual Information (MI) has attracted attention in bounding th...
research
08/03/2023

Feature Noise Boosts DNN Generalization under Label Noise

The presence of label noise in the training data has a profound impact o...
research
10/26/2020

Examining the causal structures of deep neural networks using information theory

Deep Neural Networks (DNNs) are often examined at the level of their res...
research
05/13/2023

Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression

The Information Bottleneck (IB) principle offers an information-theoreti...
research
05/05/2021

A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs

This paper focuses on understanding how the generalization error scales ...
research
01/04/2022

On the Minimal Adversarial Perturbation for Deep Neural Networks with Provable Estimation Error

Although Deep Neural Networks (DNNs) have shown incredible performance i...

Please sign up or login with your details

Forgot password? Click here to reset