Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions

01/08/2019
by   Anna Sergeevna Bosman, et al.
0

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points via gradient-based random sampling. The proposed technique is used to perform an empirical study of the loss surfaces generated by two different error metrics: quadratic loss and entropic loss. The empirical observations confirm the theoretical hypothesis regarding the nature of neural network attraction basins. Entropic loss is shown to exhibit stronger gradients and fewer stationary points than quadratic loss, indicating that entropic loss has a more searchable landscape. Quadratic loss is shown to be more resilient to overfitting than entropic loss. Both losses are shown to exhibit local minima, but the number of local minima is shown to decrease with an increase in dimensionality. Thus, the proposed visualisation technique successfully captures the local minima properties exhibited by the neural network loss surfaces, and can be used for the purpose of fitness landscape analysis of neural networks.

READ FULL TEXT
research
04/26/2022

Hybridised Loss Functions for Improved Neural Network Generalisation

Loss functions play an important role in the training of artificial neur...
research
05/24/2019

Loss Surface Modality of Feed-Forward Neural Network Architectures

It has been argued in the past that high-dimensional neural networks do ...
research
11/29/2019

Barcodes as summary of objective function's topology

We apply the canonical forms (barcodes) of gradient Morse complexes to e...
research
04/06/2018

The Loss Surface of XOR Artificial Neural Networks

Training an artificial neural network involves an optimization process o...
research
12/31/2020

Topological obstructions in neural networks learning

We apply methods of topological data analysis to loss functions to gain ...
research
06/18/2018

Using Mode Connectivity for Loss Landscape Analysis

Mode connectivity is a recently introduced frame- work that empirically ...
research
03/03/2018

On the Power of Over-parametrization in Neural Networks with Quadratic Activation

We provide new theoretical insights on why over-parametrization is effec...

Please sign up or login with your details

Forgot password? Click here to reset