Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization

11/09/2022
by   Alexander Wikner, et al.
0

Recent work has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of unknown chaotic dynamical systems. Such ML models can be used to produce both short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (“climate”). Both of these tasks can be accomplished by employing a feedback loop, whereby the model is trained to predict forward one time step, then the trained model is iterated for multiple time steps with its output used as the input. In the absence of mitigating techniques, however, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. In this article, we systematically examine the technique of adding noise to the ML model input during training as a means to promote stability and improve prediction accuracy. Furthermore, we introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. Our case study uses reservoir computing, a machine-learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while reservoir computers trained without regularization are unstable. Compared with other types of regularization that yield stability in some cases, we find that both short-term and climate predictions from reservoir computers trained with noise or with LMNT are substantially more accurate. Finally, we show that the deterministic aspect of our LMNT regularization facilitates fast hyperparameter tuning when compared to training with noise.

READ FULL TEXT

page 25

page 26

page 30

research
10/01/2019

Forecasting Chaotic Systems with Very Low Connectivity Reservoir Computers

We explore the hyperparameter space of reservoir computers used for fore...
research
11/15/2022

Emergence of a stochastic resonance in machine learning

Can noise be beneficial to machine-learning prediction of chaotic system...
research
09/19/2023

Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling

Accurate long-term predictions are the foundations for many machine lear...
research
09/11/2023

Learning noise-induced transitions by multi-scaling reservoir computing

Noise is usually regarded as adversarial to extract the effective dynami...
research
12/05/2019

Using Machine Learning to Assess Short Term Causal Dependence and Infer Network Links

We introduce and test a general machine-learning-based technique for the...
research
04/28/2023

Temporal Subsampling Diminishes Small Spatial Scales in Recurrent Neural Network Emulators of Geophysical Turbulence

The immense computational cost of traditional numerical weather and clim...
research
09/17/2018

Learning short-term past as predictor of human behavior in commercial buildings

This paper addresses the question of identifying the time-window in shor...

Please sign up or login with your details

Forgot password? Click here to reset