A Neural Scaling Law from the Dimension of the Data Manifold

04/22/2020
by   Utkarsh Sharma, et al.
0

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law L ∝ N^-α in the number of network parameters N. This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension d. This simple theory predicts that the scaling exponents α≈ 4/d for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of d and α by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/23/2020

Scaling Laws for Neural Language Models

We study empirical scaling laws for language model performance on the cr...
research
03/23/2023

The Quantization Model of Neural Scaling

We propose the Quantization Model of neural scaling laws, explaining bot...
research
02/12/2021

Explaining Neural Scaling Laws

The test loss of well-trained neural networks often follows precise powe...
research
01/31/2023

Scaling laws for single-agent reinforcement learning

Recent work has shown that, in generative modeling, cross-entropy loss i...
research
06/18/2020

On the Predictability of Pruning Across Scales

We show that the error of magnitude-pruned networks follows a scaling la...
research
09/16/2021

Scaling Laws for Neural Machine Translation

We present an empirical study of scaling properties of encoder-decoder T...
research
07/21/2017

Ultraslow diffusion in language: Dynamics of appearance of already popular adjectives on Japanese blogs

What dynamics govern a time series representing the appearance of words ...

Please sign up or login with your details

Forgot password? Click here to reset