DeepAI AI Chat
Log In Sign Up

Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

by   Wesley J. Maddox, et al.

Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity. Indeed, neural networks often have many more parameters than there are data points, yet still provide good generalization performance. Moreover, when we measure generalization as a function of parameters, we see double descent behaviour, where the test error decreases, increases, and then again decreases. We show that many of these properties become understandable when viewed through the lens of effective dimensionality, which measures the dimensionality of the parameter space determined by the data. We relate effective dimensionality to posterior contraction in Bayesian deep learning, model selection, double descent, and functional diversity in loss surfaces, leading to a richer understanding of the interplay between parameters and functions in deep models.


page 2

page 7

page 15


Generalization Through The Lens Of Leave-One-Out Error

Despite the tremendous empirical success of deep learning models to solv...

Deep Double Descent: Where Bigger Models and More Data Hurt

We show that a variety of modern deep learning tasks exhibit a "double-d...

When and how epochwise double descent happens

Deep neural networks are known to exhibit a `double descent' behavior as...

Learning Capacity: A Measure of the Effective Dimensionality of a Model

We exploit a formal correspondence between thermodynamics and inference,...

Numerical Exploration of Training Loss Level-Sets in Deep Neural Networks

We present a computational method for empirically characterizing the tra...

Phenomenology of Double Descent in Finite-Width Neural Networks

`Double descent' delineates the generalization behaviour of models depen...

An Effective Approach to Minimize Error in Midpoint Ellipse Drawing Algorithm

The present paper deals with the generalization of Midpoint Ellipse Draw...

Code Repositories


Public Codebase for Rethinking Parameter Counting: Effective Dimensionality Revisited

view repo