Learning from Randomly Initialized Neural Network Features

02/13/2022
by   Ehsan Amid, et al.
0

We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which is inherently infinite-dimensional. We conduct ablations across multiple architectures of varying sizes as well as initializations and activation functions. Our analysis suggests that certain structures that manifest in a trained model are already present at initialization. Therefore, NNPK may provide further insight into why neural networks are so effective in learning such structures.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset