Empirical Strategy for Stretching Probability Distribution in Neural-network-based Regression

09/08/2020
by   Eunho Koo, et al.
0

In regression analysis under artificial neural networks, the prediction performance depends on determining the appropriate weights between layers. As randomly initialized weights are updated during back-propagation using the gradient descent procedure under a given loss function, the loss function structure can affect the performance significantly. In this study, we considered the distribution error, i.e., the inconsistency of two distributions (those of the predicted values and label), as the prediction error, and proposed weighted empirical stretching (WES) as a novel loss function to increase the overlap area of the two distributions. The function depends on the distribution of a given label, thus, it is applicable to any distribution shape. Moreover, it contains a scaling hyperparameter such that the appropriate parameter value maximizes the common section of the two distributions. To test the function capability, we generated ideal distributed curves (unimodal, skewed unimodal, bimodal, and skewed bimodal) as the labels, and used the Fourier-extracted input data from the curves under a feedforward neural network. In general, WES outperformed loss functions in wide use, and the performance was robust to the various noise levels. The improved results in RMSE for the extreme domain (i.e., both tail regions of the distribution) are expected to be utilized for prediction of abnormal events in non-linear complex systems such as natural disaster and financial crisis.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/21/2018

Wrapped Loss Function for Regularizing Nonconforming Residual Distributions

Multi-output is essential in machine learning that it might suffer from ...
research
11/26/2019

Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks

Motivated by the flexibility of biological neural networks whose connect...
research
08/15/2020

Correspondence between neuroevolution and gradient descent

We show analytically that training a neural network by stochastic mutati...
research
09/30/2019

Towards Robust Direct Perception Networks for Automated Driving

We consider the problem of engineering robust direct perception neural n...
research
07/23/2020

Adma: A Flexible Loss Function for Neural Networks

Highly increased interest in Artificial Neural Networks (ANNs) have resu...
research
06/05/2023

On Tail Decay Rate Estimation of Loss Function Distributions

The study of loss function distributions is critical to characterize a m...
research
09/05/2022

Full Kullback-Leibler-Divergence Loss for Hyperparameter-free Label Distribution Learning

The concept of Label Distribution Learning (LDL) is a technique to stabi...

Please sign up or login with your details

Forgot password? Click here to reset