Effective Regularization Through Loss-Function Metalearning

10/02/2020
by   Santiago Gonzalez, et al.
0

Loss-function metalearning can be used to discover novel, customized loss functions for deep neural networks, resulting in improved performance, faster training, and improved data utilization. A likely explanation is that such functions discourage overfitting, leading to effective regularization. This paper theoretically demonstrates that this is indeed the case: decomposition of learning rules makes it possible to characterize the training dynamics and show that loss functions evolved through TaylorGLO regularize both in the beginning and end of learning, and maintain an invariant in between. The invariant can be utilized to make the metalearning process more efficient in practice, and the regularization can train networks that are robust against adversarial attacks. Loss-function optimization can thus be seen as a well-founded new aspect of metalearning in neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/31/2020

Evolving Loss Functions With Multivariate Taylor Polynomial Parameterizations

Loss function optimization for neural networks has recently emerged as a...
research
07/28/2023

How regularization affects the geometry of loss functions

What neural networks learn depends fundamentally on the geometry of the ...
research
07/23/2020

Adma: A Flexible Loss Function for Neural Networks

Highly increased interest in Artificial Neural Networks (ANNs) have resu...
research
06/28/2019

Learning Effective Loss Functions Efficiently

We consider the problem of learning a loss function which, when minimize...
research
08/27/2020

Adversarially Robust Learning via Entropic Regularization

In this paper we propose a new family of algorithms for training adversa...
research
07/16/2021

Entropic alternatives to initialization

Local entropic loss functions provide a versatile framework to define ar...
research
07/21/2021

Memorization in Deep Neural Networks: Does the Loss Function matter?

Deep Neural Networks, often owing to the overparameterization, are shown...

Please sign up or login with your details

Forgot password? Click here to reset