Global Convergence of Sobolev Training for Overparametrized Neural Networks

06/14/2020
by   Jorio Cocola, et al.
0

Sobolev loss is used when training a network to approximate the values and derivatives of a target function at a prescribed set of input points. Recent works have demonstrated its successful applications in various tasks such as distillation or synthetic gradient prediction. In this work we prove that an overparametrized two-layer relu neural network trained on the Sobolev loss with gradient flow from random initialization can fit any given function values and any given directional derivatives, under a separation condition on the input data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/07/2022

Adversarial Reprogramming Revisited

Adversarial reprogramming, introduced by Elsayed, Goodfellow, and Sohl-D...
research
05/15/2023

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

Rectified linear unit (ReLU), as a non-linear activation function, is we...
research
07/09/2020

Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK

We consider the dynamic of gradient descent for learning a two-layer neu...
research
02/09/2021

When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?

We establish conditions under which gradient descent applied to fixed-wi...
research
12/05/2022

Improved Convergence Guarantees for Shallow Neural Networks

We continue a long line of research aimed at proving convergence of dept...
research
07/24/2023

Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization

This paper studies the problem of training a two-layer ReLU network for ...
research
01/24/2019

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

Recent works have cast some light on the mystery of why deep nets fit an...

Please sign up or login with your details

Forgot password? Click here to reset