Log In Sign Up

Visualizing High-Dimensional Trajectories on the Loss-Landscape of ANNs

by   Stefan Horoi, et al.

Training artificial neural networks requires the optimization of highly non-convex loss functions. Throughout the years, the scientific community has developed an extensive set of tools and architectures that render this optimization task tractable and a general intuition has been developed for choosing hyper parameters that help the models reach minima that generalize well to unseen data. However, for the most part, the difference in trainability in between architectures, tasks and even the gap in network generalization abilities still remain unexplained. Visualization tools have played a key role in uncovering key geometric characteristics of the loss-landscape of ANNs and how they impact trainability and generalization capabilities. However, most visualizations methods proposed so far have been relatively limited in their capabilities since they are of linear nature and only capture features in a limited number of dimensions. We propose the use of the modern dimensionality reduction method PHATE which represents the SOTA in terms of capturing both global and local structures of high-dimensional data. We apply this method to visualize the loss landscape during and after training. Our visualizations reveal differences in training trajectories and generalization capabilities when used to make comparisons between optimization methods, initializations, architectures, and datasets. Given this success we anticipate this method to be used in making informed choices about these aspects of neural networks.


page 1

page 2

page 3

page 4


Visualizing the Loss Landscape of Neural Nets

Neural network training relies on our ability to find "good" minimizers ...

Visualizing the PHATE of Neural Networks

Understanding why and how certain neural networks outperform others is k...

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

It is widely observed that deep learning models with learned parameters ...

FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks

Despite their effective use in various fields, many aspects of neural ne...

Understanding Generalization through Visualizations

The power of neural networks lies in their ability to generalize to unse...

The Loss Surface of XOR Artificial Neural Networks

Training an artificial neural network involves an optimization process o...

Visualizing the geometry of labeled high-dimensional data with spheres

Data visualizations summarize high-dimensional distributions in two or t...