Convergence Analysis of the Dynamics of a Special Kind of Two-Layered Neural Networks with ℓ_1 and ℓ_2 Regularization

11/19/2017
by   Zhifeng Kong, et al.
0

In this paper, we made an extension to the convergence analysis of the dynamics of two-layered bias-free networks with one ReLU output. We took into consideration two popular regularization terms: the ℓ_1 and ℓ_2 norm of the parameter vector w, and added it to the square loss function with coefficient λ/2. We proved that when λ is small, the weight vector w converges to the optimal solution ŵ (with respect to the new loss function) with probability ≥ (1-ε)(1-A_d)/2 under random initiations in a sphere centered at the origin, where ε is a small value and A_d is a constant. Numerical experiments including phase diagrams and repeated simulations verified our theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/07/2019

Only sparsity based loss function for learning representations

We study the emergence of sparse representations in neural networks. We ...
research
07/28/2023

How regularization affects the geometry of loss functions

What neural networks learn depends fundamentally on the geometry of the ...
research
06/24/2023

Current density impedance imaging with PINNs

In this paper, we introduce CDII-PINNs, a computationally efficient meth...
research
07/16/2020

Understanding Implicit Regularization in Over-Parameterized Nonlinear Statistical Model

We study the implicit regularization phenomenon induced by simple optimi...
research
01/06/2023

Sensitivity analysis using Physics-informed neural networks

The paper's goal is to provide a simple unified approach to perform sens...
research
04/15/2020

Towards a theory of machine learning

We define a neural network as a septuple consisting of (1) a state vecto...
research
06/25/2020

Implicitly Maximizing Margins with the Hinge Loss

A new loss function is proposed for neural networks on classification ta...

Please sign up or login with your details

Forgot password? Click here to reset