SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions

12/27/2019
by   A. Ali Heydari, et al.
58

Adaptive loss function formulation is an active area of research and has gained a great deal of popularity in recent years, following the success of deep learning. However, existing frameworks of adaptive loss functions often suffer from slow convergence and poor choice of weights for the loss components. Traditionally, the elements of a multi-part loss function are weighted equally or their weights are determined through heuristic approaches that yield near-optimal (or sub-optimal) results. To address this problem, we propose a family of methods, called SoftAdapt, that dynamically change function weights for multi-part loss functions based on live performance statistics of the component losses. SoftAdapt is mathematically intuitive, computationally efficient and straightforward to implement. In this paper, we present the mathematical formulation and pseudocode for SoftAdapt, along with results from applying our methods to image reconstruction (Sparse Autoencoders) and synthetic data generation (Introspective Variational Autoencoders).

READ FULL TEXT

page 1

page 6

page 8

research
02/25/2019

Matrix denoising for weighted loss functions and heterogeneous signals

We consider the problem of recovering a low-rank matrix from a noisy obs...
research
10/28/2022

Evaluating the Impact of Loss Function Variation in Deep Learning for Classification

The loss function is arguably among the most important hyperparameters f...
research
05/17/2019

AM-LFS: AutoML for Loss Function Search

Designing an effective loss function plays an important role in visual a...
research
08/28/2021

Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statistics

We propose a generalized formulation of the Huber loss. We show that wit...
research
12/05/2020

Adaptive Weighted Discriminator for Training Generative Adversarial Networks

Generative adversarial network (GAN) has become one of the most importan...
research
03/06/2023

The α-divergence Improves the Entropy Production Estimation via Machine Learning

Recent years have seen a surge of interest in the algorithmic estimation...
research
01/20/2021

Component Tree Loss Function: Definition and Optimization

In this article, we propose a method to design loss functions based on c...

Please sign up or login with your details

Forgot password? Click here to reset