On Symmetry and Initialization for Neural Networks

07/01/2019
by   Ido Nachum, et al.
0

This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/30/2022

On the optimization and generalization of overparameterized implicit neural networks

Implicit neural networks have become increasingly attractive in the mach...
research
06/01/2023

Initial Guessing Bias: How Untrained Networks Favor Some Classes

The initial state of neural networks plays a central role in conditionin...
research
08/16/2020

A Functional Perspective on Learning Symmetric Functions with Neural Networks

Symmetric functions, which take as input an unordered, fixed-size set, a...
research
04/21/2021

Deep limits and cut-off phenomena for neural networks

We consider dynamical and geometrical aspects of deep learning. For many...
research
03/12/2023

Phase Diagram of Initial Condensation for Two-layer Neural Networks

The phenomenon of distinct behaviors exhibited by neural networks under ...
research
02/18/2016

Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

We develop a general duality between neural networks and compositional k...
research
02/24/2020

TrojanNet: Embedding Hidden Trojan Horse Models in Neural Networks

The complexity of large-scale neural networks can lead to poor understan...

Please sign up or login with your details

Forgot password? Click here to reset