Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective

by   Gowthami Somepalli, et al.

We discuss methods for visualizing neural network decision boundaries and decision regions. We use these visualizations to investigate issues related to reproducibility and generalization in neural network training. We observe that changes in model architecture (and its associate inductive bias) cause visible changes in decision boundaries, while multiple runs with the same architecture yield results with strong similarities, especially in the case of wide architectures. We also use decision boundary methods to visualize double descent phenomena. We see that decision boundary reproducibility depends strongly on model width. Near the threshold of interpolation, neural network decision boundaries become fragmented into many small decision regions, and these regions are non-reproducible. Meanwhile, very narrows and very wide networks have high levels of reproducibility in their decision boundaries with relatively few decision regions. We discuss how our observations relate to the theory of double descent phenomena in convex models. Code is available at


page 1

page 2

page 3

page 4

page 7

page 12

page 13

page 14


The Vanishing Decision Boundary Complexity and the Strong First Component

We show that unlike machine learning classifiers, there are no complex b...

On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective

This work tackles the problem of characterizing and understanding the de...

Multiphysics discovery with moving boundaries using Ensemble SINDy and Peridynamic Differential Operator

This study proposes a novel framework for learning the underlying physic...

Neural Tangent Kernel Eigenvalues Accurately Predict Generalization

Finding a quantitative theory of neural network generalization has long ...

Neural networks learn to magnify areas near decision boundaries

We study how training molds the Riemannian geometry induced by neural ne...

Border Effect of Complex Network: An analysis on the cooperation network of movie stars across different regions

People always carry out social activities within certain geographic spac...

Please sign up or login with your details

Forgot password? Click here to reset