DeepAI AI Chat
Log In Sign Up

Identity Crisis: Memorization and Generalization under Extreme Overparameterization

02/13/2019
by   Chiyuan Zhang, et al.
Google
berkeley college
26

We study the interplay between memorization and generalization of overparametrized networks in the extreme case of a single training example. The learning task is to predict an output which is as similar as possible to the input. We examine both fully-connected and convolutional networks that are initialized randomly and then trained to minimize the reconstruction error. The trained networks take one of the two forms: the constant function ("memorization") and the identity function ("generalization"). We show that different architectures exhibit vastly different inductive bias towards memorization and generalization. An important consequence of our study is that even in extreme cases of overparameterization, deep learning can result in proper generalization.

READ FULL TEXT

page 7

page 10

page 20

page 22

page 23

page 25

page 26

page 28

10/16/2020

Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?

Convolutional neural networks often dominate fully-connected counterpart...
08/24/2022

On a Built-in Conflict between Deep Learning and Systematic Generalization

In this paper, we hypothesize that internal function sharing is one of t...
06/09/2022

Redundancy in Deep Linear Neural Networks

Conventional wisdom states that deep linear neural networks benefit from...
10/24/2022

A PAC-Bayesian Generalization Bound for Equivariant Networks

Equivariant networks capture the inductive bias about the symmetry of th...
02/23/2018

Sensitivity and Generalization in Neural Networks: an Empirical Study

In practice it is often found that large over-parameterized neural netwo...
07/30/2020

Generalization Comparison of Deep Neural Networks via Output Sensitivity

Although recent works have brought some insights into the performance im...
02/14/2023

The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs

Classification margins are commonly used to estimate the generalization ...