Tilting the playing field: Dynamical loss functions for machine learning

by   Miguel Ruiz-Garcia, et al.

We show that learning can be improved by using loss functions that evolve cyclically during training to emphasize one class at a time. In underparameterized networks, such dynamical loss functions can lead to successful training for networks that fail to find a deep minima of the standard cross-entropy loss. In overparameterized networks, dynamical loss functions can lead to better generalization. Improvement arises from the interplay of the changing loss landscape with the dynamics of the system as it evolves to minimize the loss. In particular, as the loss function oscillates, instabilities develop in the form of bifurcation cascades, which we study using the Hessian and Neural Tangent Kernel. Valleys in the landscape widen and deepen, and then narrow and rise as the loss landscape changes during a cycle. As the landscape narrows, the learning rate becomes too large and the network becomes unstable and bounces around the valley. This process ultimately pushes the system into deeper and wider regions of the loss landscape and is characterized by decreasing eigenvalues of the Hessian. This results in better regularized models with improved generalization performance.



There are no comments yet.


page 3

page 4


Visualizing the Loss Landscape of Neural Nets

Neural network training relies on our ability to find "good" minimizers ...

Extracting Global Dynamics of Loss Landscape in Deep Learning Models

Deep learning models evolve through training to learn the manifold in wh...

Robustness of different loss functions and their impact on networks learning capability

Recent developments in AI have made it ubiquitous, every industry is try...

Characterization of Gradient Dominance and Regularity Conditions for Neural Networks

The past decade has witnessed a successful application of deep learning ...

On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

We analyze the influence of adversarial training on the loss landscape o...

On the alpha-loss Landscape in the Logistic Model

We analyze the optimization landscape of a recently introduced tunable c...

Visualizing the Loss Landscape of Winning Lottery Tickets

The underlying loss landscapes of deep neural networks have a great impa...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.