DeepAI AI Chat
Log In Sign Up

Identity Crisis: Memorization and Generalization under Extreme Overparameterization

by   Chiyuan Zhang, et al.
berkeley college

We study the interplay between memorization and generalization of overparametrized networks in the extreme case of a single training example. The learning task is to predict an output which is as similar as possible to the input. We examine both fully-connected and convolutional networks that are initialized randomly and then trained to minimize the reconstruction error. The trained networks take one of the two forms: the constant function ("memorization") and the identity function ("generalization"). We show that different architectures exhibit vastly different inductive bias towards memorization and generalization. An important consequence of our study is that even in extreme cases of overparameterization, deep learning can result in proper generalization.


page 7

page 10

page 20

page 22

page 23

page 25

page 26

page 28


Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?

Convolutional neural networks often dominate fully-connected counterpart...

On a Built-in Conflict between Deep Learning and Systematic Generalization

In this paper, we hypothesize that internal function sharing is one of t...

Redundancy in Deep Linear Neural Networks

Conventional wisdom states that deep linear neural networks benefit from...

A PAC-Bayesian Generalization Bound for Equivariant Networks

Equivariant networks capture the inductive bias about the symmetry of th...

Sensitivity and Generalization in Neural Networks: an Empirical Study

In practice it is often found that large over-parameterized neural netwo...

Generalization Comparison of Deep Neural Networks via Output Sensitivity

Although recent works have brought some insights into the performance im...

The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs

Classification margins are commonly used to estimate the generalization ...