Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory

10/01/2019

∙

by   Micah Goldblum, et al.

∙

0

∙

share

We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. We study the prevalence of local minima in loss landscapes, whether small-norm parameter vectors generalize better (and whether this explains the advantages of weight decay), whether wide-network theories (like the neural tangent kernel) describe the behaviors of classifiers, and whether the rank of weight matrices can be linked to generalization and robustness in real-world networks.