Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes

by   Manuel Haussmann, et al.

We propose a novel scheme for fitting heavily parameterized non-linear stochastic differential equations (SDEs). We assign a prior on the parameters of the SDE drift and diffusion functions to achieve a Bayesian model. We then infer this model using the well-known local reparameterized trick for the first time for empirical Bayes, i.e. to integrate out the SDE parameters. The model is then fit by maximizing the likelihood of the resultant marginal with respect to a potentially large number of hyperparameters, which prohibits stable training. As the prior parameters are marginalized, the model also no longer provides a principled means to incorporate prior knowledge. We overcome both of these drawbacks by deriving a training loss that comprises the marginal likelihood of the predictor and a PAC-Bayesian complexity penalty. We observe on synthetic as well as real-world time series prediction tasks that our method provides an improved model fit accompanied with favorable extrapolation properties when provided a partial description of the environment dynamics. Hence, we view the outcome as a promising attempt for building cutting-edge hybrid learning systems that effectively combine first-principle physics and data-driven approaches.



There are no comments yet.


page 21

page 22

page 23

page 24


Bayesian Prior Networks with PAC Training

We propose to train Bayesian Neural Networks (BNNs) by empirical Bayes a...

Differential Bayesian Neural Nets

Neural Ordinary Differential Equations (N-ODEs) are a powerful building ...

PAC^m-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

While the decision-theoretic optimality of the Bayesian formalism under ...

Deterministic Inference of Neural Stochastic Differential Equations

Model noise is known to have detrimental effects on neural networks, suc...

Generalization bounds for deep learning

Generalization in deep learning has been the topic of much recent theore...

Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning

We study an approach to learning pruning masks by optimizing the expecte...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.