CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences – Version 1

by   Imme Ebert-Uphoff, et al.

Neural networks are increasingly used in environmental science applications. Furthermore, neural network models are trained by minimizing a loss function, and it is crucial to choose the loss function very carefully for environmental science applications, as it determines what exactly is being optimized. Standard loss functions do not cover all the needs of the environmental sciences, which makes it important for scientists to be able to develop their own custom loss functions so that they can implement many of the classic performance measures already developed in environmental science, including measures developed for spatial model verification. However, there are very few resources available that cover the basics of custom loss function development comprehensively, and to the best of our knowledge none that focus on the needs of environmental scientists. This document seeks to fill this gap by providing a guide on how to write custom loss functions targeted toward environmental science applications. Topics include the basics of writing custom loss functions, common pitfalls, functions to use in loss functions, examples such as fractions skill score as loss function, how to incorporate physical constraints, discrete and soft discretization, and concepts such as focal, robust, and adaptive loss. While examples are currently provided in this guide for Python with Keras and the TensorFlow backend, the basic concepts also apply to other environments, such as Python with PyTorch. Similarly, while the sample loss functions provided here are from meteorology, these are just examples of how to create custom loss functions. Other fields in the environmental sciences have very similar needs for custom loss functions, e.g., for evaluating spatial forecasts effectively, and the concepts discussed here can be applied there as well. All code samples are provided in a GitHub repository.


page 15

page 16

page 21

page 23

page 24

page 26

page 29

page 32


Can we integrate spatial verification methods into neural-network loss functions for atmospheric science?

In the last decade, much work in atmospheric science has focused on spat...

Improved Training Speed, Accuracy, and Data Utilization Through Loss Function Optimization

As the complexity of neural network models has grown, it has become incr...

Effective Regularization Through Loss-Function Metalearning

Loss-function metalearning can be used to discover novel, customized los...

Improving Computer Generated Dialog with Auxiliary Loss Functions and Custom Evaluation Metrics

Although people have the ability to engage in vapid dialogue without eff...

Alpha-Net: Architecture, Models, and Applications

Deep learning network training is usually computationally expensive and ...

Applying physics-based loss functions to neural networks for improved generalizability in mechanics problems

Physics-Informed Machine Learning (PIML) has gained momentum in the last...

Controlled abstention neural networks for identifying skillful predictions for classification problems

The earth system is exceedingly complex and often chaotic in nature, mak...