Inferring untrained complex dynamics of delay systems using an adapted echo state network

by   Mirko Goldmann, et al.

Caused by finite signal propagation velocities, many complex systems feature time delays that may induce high-dimensional chaotic behavior and make forecasting intricate. Here, we propose an echo state network adaptable to the physics of systems with arbitrary delays. After training the network to forecast a system with a unique and sufficiently long delay, it already learned to predict the system dynamics for all other delays. A simple adaptation of the network's topology allows us to infer untrained features such as high-dimensional chaotic attractors, bifurcations, and even multistabilities, that emerge with shorter and longer delays. Thus, the fusion of physical knowledge of the delay system and data-driven machine learning yields a model with high generalization capabilities and unprecedented prediction accuracy.



There are no comments yet.


page 4


Multi-Airport Delay Prediction with Transformers

Airport performance prediction with a reasonable look-ahead time is a ch...

Computational tool to study high dimensional dynamic in NMM

Neuroscience has shown great progress in recent years. Several of the th...

Machine Learning assisted Chimera and Solitary states in Networks

Chimera and Solitary states have captivated scientists and engineers due...

Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study

Analytic, first-principles performance modeling of distributed-memory ap...

Link inference of noisy delay-coupled networks: Machine learning and opto-electronic experimental tests

We devise a machine learning technique to solve the general problem of i...

Prediction in Projection: A new paradigm in delay-coordinate reconstruction

Delay-coordinate embedding is a powerful, time-tested mathematical frame...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


We acknowledge the Spanish State Research Agency, through the Severo Ochoa and María de Maeztu Program for Centers and Units of Excellence in R&D (MDM-2017-0711) and through the QUARESC project (PID2019-109094GB-C21 and -C22/ AEI / 10.13039/501100011033) and DECAPH project (PID2019-111537GB-C21 and -C22/ AEI / 10.13039/501100011033). M.G. acknowledges financial support by the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 860360 (POST DIGITAL). The work of MCS has been supported by MICINN/AEI/FEDER and the University of the Balearic Islands through a “Ramon y Cajal” Fellowship (RYC-2015-18140). This work has been partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 899265 (ADOPD).


  • Tang et al. (2020) Y. Tang, J. Kurths, W. Lin, E. Ott, and L. Kocarev, Introduction to focus issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics, Chaos 30, 063151 (2020).
  • Lazer et al. (2014) D. Lazer, R. Kennedy, G. King, and A. Vespignani, The parable of google flu: traps in big data analysis, Science 343, 1203 (2014).
  • Karniadakis et al. (2021) G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, Physics-informed machine learning, Nat. Rev. Phys. 3, 422 (2021).
  • Elmarakeby et al. (2021) H. A. Elmarakeby, J. Hwang, R. Arafeh, J. Crowdis, S. Gang, D. Liu, S. H. AlDubayan, K. Salari, S. Kregel, C. Richter, et al., Biologically informed deep neural network for prostate cancer discovery, Nature 598, 348–352 (2021).
  • Lukoševičius et al. (2012) M. Lukoševičius, H. Jaeger, and B. Schrauwen, Reservoir computing trends, KI-Künstliche Intelligenz 26, 365 (2012).
  • Jaeger (2001)

    H. Jaeger, The “echo state” approach to analysing and training recurrent neural networks-with an erratum note, Bonn, Germany: German National Research Center for Information Technology GMD Technical Report 

    148, 13 (2001).
  • Maass et al. (2002) W. Maass, T. Natschläger, and H. Markram, Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput. 14, 2531 (2002).
  • Nakajima and Fischer (2021) K. Nakajima and I. Fischer, eds., Reservoir Computing: Theory, Physical Implementations, and Applications, Natural Computing Series (Springer, Singapore, 2021).
  • Jaeger and Haas (2004) H. Jaeger and H. Haas, Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication, Science 304, 78 (2004).
  • Pathak et al. (2017) J. Pathak, Z. Lu, B. R. Hunt, M. Girvan, and E. Ott, Using machine learning to replicate chaotic attractors and calculate lyapunov exponents from data, Chaos 27, 121102 (2017).
  • Pathak et al. (2018a) J. Pathak, B. Hunt, M. Girvan, Z. Lu, and E. Ott, Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Phys. Rev. Lett. 120, 024102 (2018a).
  • Klos et al. (2020) C. Klos, Y. F. K. Kossio, S. Goedeke, A. Gilra, and R.-M. Memmesheimer, Dynamical learning of dynamics, Phys. Rev. Lett. 125, 088103 (2020).
  • Röhm et al. (2021) A. Röhm, D. J. Gauthier, and I. Fischer, Model-free inference of unseen attractors: Reconstructing phase space features from a single noisy trajectory using reservoir computing, Chaos 31, 103127 (2021).
  • Antonik et al. (2018) P. Antonik, M. Gulina, J. Pauwels, and S. Massar, Using a reservoir computer to learn chaotic attractors, with applications to chaos synchronization and cryptography, Phys. Rev. E 98, 1 (2018)1802.02844 .
  • Lu et al. (2018) Z. Lu, B. R. Hunt, and E. Ott, Attractor reconstruction by machine learning, Chaos 28, 061104 (2018).
  • Pathak et al. (2018b) J. Pathak, A. Wikner, R. Fussell, S. Chandra, B. R. Hunt, M. Girvan, and E. Ott, Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model, Chaos 28, 041101 (2018b).
  • Erneux (2009) T. Erneux, Applied delay differential equations, Vol. 3 (Springer Science & Business Media, 2009).
  • Fischer et al. (1994) I. Fischer, O. Hess, W. Elsäer, and E. Göbel, High-dimensional chaotic dynamics of an external cavity semiconductor laser, Phys. Rev. Lett. 73, 2188 (1994).
  • Le Berre et al. (1986) M. Le Berre, E. Ressayre, A. Tallet, and H. M. Gibbs, High-dimension chaotic attractors of a nonlinear ring cavity, Phys. Rev. Lett. 56, 274 (1986).
  • Heiligenthal et al. (2011) S. Heiligenthal, T. Dahms, S. Yanchuk, T. Jüngling, V. Flunkert, I. Kanter, E. Schöll, and W. Kinzel, Strong and weak chaos in nonlinear networks with time-delayed couplings, Phys. Rev. Lett. 107, 234102 (2011).
  • Sethia et al. (2008) G. C. Sethia, A. Sen, and F. M. Atay, Clustered chimera states in delay-coupled oscillator systems, Phys. Rev. Lett. 100, 144102 (2008).
  • Hegger et al. (1998) R. Hegger, M. J. Bünner, H. Kantz, and A. Giaquinta, Identifying and modeling delay feedback systems, Phys. Rev. Lett. 81, 558 (1998).
  • Penkovsky et al. (2019)

    B. Penkovsky, X. Porte, M. Jacquot, L. Larger, and D. Brunner, Coupled nonlinear delay systems as deep convolutional neural networks

    Phys. Rev. Lett. 123, 054101 (2019).
  • Antonik et al. (2017) P. Antonik, M. Haelterman, and S. Massar, Brain-inspired photonic signal processor for generating periodic patterns and emulating chaotic systems, Phys. Rev. Appl. 7, 054014 (2017).
  • Goldmann et al. (2020) M. Goldmann, F. Köster, K. Lüdge, and S. Yanchuk, Deep time-delay reservoir computing: Dynamics and memory capacity, Chaos 30, 093124 (2020).
  • Chen et al. (2020) X. Chen, T. Weng, H. Yang, C. Gu, J. Zhang, and M. Small, Mapping topological characteristics of dynamical systems into neural networks: A reservoir computing approach, Phys. Rev. E 102, 033314 (2020).
  • Marquez et al. (2019)

    B. A. Marquez, J. Suarez-Vargas, and B. J. Shastri, Takens-inspired neuromorphic processor: A downsizing tool for random recurrent neural networks via feature extraction

    arXiv 1, 33030 (2019)1907.03122 .
  • Zhu et al. (2019) Q. Zhu, H. Ma, and W. Lin, Detecting unstable periodic orbits based only on time series: When adaptive delayed feedback control meets reservoir computing, Chaos 29, 093125 (2019).
  • Haluszczynski and Räth (2019) A. Haluszczynski and C. Räth, Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing, Chaos 2910.1063/1.5118725 (2019), 1907.05639 .
  • (30) See Supplemental Material at [URL will be inserted by publisher] for the results of the dESN performing the closed-loop mode continuation of the Mackey-Glass system with and , respectively. Further, we explain how to implement a delay-based reservoir such that it has similar capabilities as the dESN. Finally, we show the results of training a dESN to predict the bifurcation diagram of an Ikeda delay system.Larger et al. (2013); Erneux et al. (2004).
  • Mackey and Glass (1977) M. C. Mackey and L. Glass, Oscillation and chaos in physiological control systems, Science 197, 287 (1977).
  • Wang et al. (2016) W.-X. Wang, Y.-C. Lai, and C. Grebogi, Data based identification and prediction of nonlinear and complex dynamical systems, Phys. Rep. 644, 1 (2016).
  • Ma et al. (2013) H. Ma, W. Lin, and Y.-C. Lai, Detecting unstable periodic orbits in high-dimensional chaotic systems from time series: Reconstruction meeting with adaptation, Phys. Rev. E 87, 050901 (2013).
  • Appeltant et al. (2011) L. Appeltant, M. C. Soriano, G. Van Der Sande, J. Danckaert, S. Massar, J. Dambre, B. Schrauwen, C. R. Mirasso, and I. Fischer, Information processing using a single dynamical node as complex system, Nat. Commun. 2, 466 (2011).
  • Köster et al. (2020) F. Köster, D. Ehlert, and K. Lüdge, Limitations of the recall capabilities in delay-based reservoir computing systems, Cognit Comput , 1 (2020).
  • Pesin (1977) J. Pesin, Characteristic lyapunov exponents and smooth ergodic theory, Russ. Math Surveys 32, 55 (1977).
  • Vicente et al. (2005) R. Vicente, J. Daudén, P. Colet, and R. Toral, Analysis and characterization of the hyperchaos generated by a semiconductor laser subject to a delayed feedback loop, IEEE J. Quantum Electron. 41, 541 (2005).
  • Farmer (1982) J. D. Farmer, Chaotic attractors of an infinite-dimensional dynamical system, Physica D 4, 366 (1982).
  • Kim et al. (2021) J. Z. Kim, Z. Lu, E. Nozari, G. J. Pappas, and D. S. Bassett, Teaching recurrent neural networks to infer global temporal structure from local examples, Nat. Mach. Intell. 3, 316 (2021).
  • Kong et al. (2021a) L.-W. Kong, H. Fan, C. Grebogi, and Y.-C. Lai, Emergence of transient chaos and intermittency in machine learning, JPhys Complexity 2, 035014 (2021a).
  • Kong et al. (2021b) L.-W. Kong, H.-W. Fan, C. Grebogi, and Y.-C. Lai, Machine learning prediction of critical transition and system collapse, Phys. Rev. Res. 3, 13090 (2021b)2012.01545 .
  • Larger et al. (2013) L. Larger, B. Penkovsky, and Y. Maistrenko, Virtual chimera states for delayed-feedback systems, Phys. Rev. Lett. 111, 054103 (2013).
  • Erneux et al. (2004) T. Erneux, L. Larger, M. W. Lee, and J.-P. Goedgebuer, Ikeda hopf bifurcation revisited, Physica D 194, 49 (2004).