Generalization Bounds with Data-dependent Fractal Dimensions

02/06/2023
by   Benjamin Dupuis, et al.
0

Providing generalization guarantees for modern neural networks has been a crucial task in statistical learning. Recently, several studies have attempted to analyze the generalization error in such settings by using tools from fractal geometry. While these works have successfully introduced new mathematical tools to apprehend generalization, they heavily rely on a Lipschitz continuity assumption, which in general does not hold for neural networks and might make the bounds vacuous. In this work, we address this issue and prove fractal geometry-based generalization bounds without requiring any Lipschitz assumption. To achieve this goal, we build up on a classical covering argument in learning theory and introduce a data-dependent fractal dimension. Despite introducing a significant amount of technical complications, this new notion lets us control the generalization error (over either fixed or random hypothesis spaces) along with certain mutual information (MI) terms. To provide a clearer interpretation to the newly introduced MI terms, as a next step, we introduce a notion of "geometric stability" and link our bounds to the prior art. Finally, we make a rigorous connection between the proposed data-dependent dimension and topological data analysis tools, which then enables us to compute the dimension in a numerically efficient way. We support our theory with experiments conducted on various settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/09/2019

Hypothesis Set Stability and Generalization

We present an extensive study of generalization for data-dependent hypot...
research
11/25/2021

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Disobeying the classical wisdom of statistical learning theory, modern d...
research
07/04/2023

Generalization Guarantees via Algorithm-dependent Rademacher Complexity

Algorithm- and data-dependent generalization bounds are required to expl...
research
03/25/2022

Generalization bounds for learning under graph-dependence: A survey

Traditional statistical learning theory relies on the assumption that da...
research
05/30/2023

How Does Information Bottleneck Help Deep Learning?

Numerous deep learning algorithms have been inspired by and understood v...
research
03/04/2022

Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithms

Understanding generalization in modern machine learning settings has bee...
research
06/07/2019

Understanding Generalization through Visualizations

The power of neural networks lies in their ability to generalize to unse...

Please sign up or login with your details

Forgot password? Click here to reset