Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

by   Amit Daniely, et al.
Stanford University

We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.


page 1

page 2

page 3

page 4


Duality for Neural Networks through Reproducing Kernel Banach Spaces

Reproducing Kernel Hilbert spaces (RKHS) have been a very successful too...

On the Quality of the Initial Basin in Overspecified Neural Networks

Deep learning, in the form of artificial neural networks, has achieved r...

Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks

In this paper, we utilize a connection between compositional kernels and...

On Symmetry and Initialization for Neural Networks

This work provides an additional step in the theoretical understanding o...

On the Modeling of Error Functions as High Dimensional Landscapes for Weight Initialization in Learning Networks

Next generation deep neural networks for classification hosted on embedd...

Linear Algebra and Duality of Neural Networks

Natural for Neural networks bases, mappings, projections and metrics are...

Please sign up or login with your details

Forgot password? Click here to reset