Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines

04/29/2021
by   Frederieke Richert, et al.
0

Over-parametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of over-parametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For on-line learning of a two-layer soft committee machine in the over-realizable case, we find that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/24/2023

Online Learning for the Random Feature Model in the Student-Teacher Framework

Deep neural networks are widely used prediction algorithms whose perform...
research
11/12/2015

Representational Distance Learning for Deep Neural Networks

Deep neural networks (DNNs) provide useful models of visual representati...
research
12/19/2014

FitNets: Hints for Thin Deep Nets

While depth tends to improve network performances, it also makes gradien...
research
03/23/2020

Neural Networks and Polynomial Regression. Demystifying the Overparametrization Phenomena

In the context of neural network models, overparametrization refers to t...
research
02/15/2023

Spatially heterogeneous learning by a deep student machine

Despite the spectacular successes, deep neural networks (DNN) with a hug...
research
09/30/2019

Over-parameterization as a Catalyst for Better Generalization of Deep ReLU network

To analyze deep ReLU network, we adopt a student-teacher setting in whic...
research
12/25/2019

Learning performance in inverse Ising problems with sparse teacher couplings

We investigate the learning performance of the pseudolikelihood maximiza...

Please sign up or login with your details

Forgot password? Click here to reset