share
Introduction
Recent advances in machine learning, optimization, and available computational resources are presenting new opportunities for data-driven modeling and simulation in the natural sciences and engineering. Empirical successes in deep learning suggest promising techniques for learning non-linear mappings and for identifying features of underlying structure
[12, 10]. Scientific computations and associated dynamical systems present a unique set of challenges for employing recent machine learning methods often motivated by image analysis and natural language processing
[2, 7, 27]. However, for scientific and engineering applications there are often important constraints required to obtain plausible models and a need for results to be more interpretable. In large-scale scientific computations modeling efforts often aim to start with first principles, or more detailed concepts, from which computation is used to obtain insights into larger-scale emergent system behaviors. Examples include the rheological responses of soft materials and complex fluids from microstructure interactions [1, 4, 18, 15], molecular dynamics modeling of protein structure and functional domains from atomic level interactions [20, 14, 6, 23], and prediction of weather and climate phenomena from detailed physical models and measurements [25, 3].Obtaining observables and quantities of interest (QoI) from simulations of such high fidelity detailed models can involve significant computational resources [19, 26, 31, 22, 21, 30]. However, many observables are not sensitive to many of the system behaviors and are expected to only depend on a subset of underlying factors. If these key features of the system could be identified for classes of observables, this would present opportunities to formulate more simplified models and make predictions which are less computationally expensive to simulate and could provide further insights into underlying mechanisms yielding system behaviors.
Recent machine learning approaches provide promising data-driven approaches for learning features and models for system behaviors from high fidelity simulation data. This includes learning data-driven models for (i) dynamics of the system at larger spatial-temporal scales (ii) interactions between system components, and (iii) features yielding coarser degrees of freedom or new quantities of interest characterizing system behaviors. The models obtained from learning can take many forms, including Deep Neural Networks (DNNs)
[10], Kernel Regression Models (KRM) [28], Gaussian Process Regression (GPR) [24], and others [11].A practical challenge is the effort often required to incorporate the contributions of such learned system features to augment existing models and simulations. Our package MLMOD aims to address this aspect of data-driven modeling by providing a general interface for incorporating ML models using standardized representations and by leveraging existing simulation frameworks such as LAMMPS. The MLMOD package provides hooks which are triggered during key parts of simulation calculations. In this way standard machine learning frameworks can be used to train ML models, such as PyTorch and TensorFlow, with the resulting models more amenable to being translated into practical simulations.
Data-Driven Modeling
Data-driven modeling can take many forms. As a specific motivation for the package and our initial prototype implementations, we discuss a rather particular example case in detail, but we aim for our overall package also to be useful in other settings. Consider the case of a detailed molecular dynamics simulation of relatively large colloidal particles within a bath of much smaller solvent particles. It is often of interest to try to infer the interaction law between the colloidal particles given the type of solution, charge, and other physical conditions. While there is extensive theoretical literature on colloidal interactions and approximate models [8, 9, 13], which have had many successes, these can also have limited accuracy and be challenging to apply in many practical settings [13, 29]. Computational simulations are widely used and allow for investigations of phenomena where detailed modeling can be employed to control for contributing physical effects.
While providing predictions related to the principles built into the detailed models, such computational simulations can be expensive given the many degrees of freedom required to represent the solvent in the entire spatial domain and from small time-scales associated with solvent-solvent interactions. However, given the size contrast of a colloid and the solvent it is expected that in many circumstances the colloidal dynamics and interactions are governed primarily through temporal averaging of the solvent-solvent and solvent-colloid interactions.
.
Relative to the detailed molecular dynamics simulation, this motivates a simplified model for the effective colloid dynamics
| (1) |
The refers to the collective configuration of all colloids.
Here the main component to determine for the simplified model is the
mobility tensor
In principle, this could be tabulated for each
by performing a sequence of computational simulations over all configurations and force combinations. However, in the general case this is inefficient in practice unless there are known symmetries or other physical structures. For example, interactions only occurring pairwise or translational and rotational invariances / equivariances. In the case of pairwise interactions, translational invariance, and rotational equivariance, the mobility can be reduced effectively to a one dimensional dependence on the configuration. In many circumstances such symmetries of the system may not be immediately apparent and even when symmetries are known tabulation can present drawbacks for interpolation and storage.
Machine learning methods provide data-driven approaches for learning representations and features for such modeling. Optimization frameworks with judicious choices of loss functions and training protocols can be used to identify system features underlying interactions, symmetries, and other structures. In machine learning methods these are represented in a compressed form over some model class allowing for efficient interpolation, and even extrapolation in some cases, especially when using explicit low dimensional latent spaces or when imposing other inductive biases.
As further simplification for the colloidal example, if we assume the interactions occur to a good approximation only pairwise the problem can be reduced to a model depending on six dimensions. This can be further constrained to learn only symmetric positive semi-definite tensors, for example by learning to generate .
There are many ways we can obtain the model
. This includes learning directly using machine learning approaches or a multi-stage approach using first a tabulation approach from more traditional estimators from statistical mechanics followed by compression to an ML model. For example, a common way to estimate mobility in fluid mechanics is to apply active forces
and compute the velocity response . The for chosen carefully. For large enough forces , the thermal fluctuations can be averaged away readily by repeating this measurement and taking the mean. In statistical mechanics, another estimator is obtained whenby using the passive fluctuations of system. A moment-based estimator commonly used is
for chosen carefully. While theoretically each of these estimators give information on , in practice there can be subtleties such as a good choice for , magnitude for , and role of fluctuations. Even for these more traditional estimators, it could still be useful for storage efficiency and convenience to train an ML model to provide a compressed representation and for interpolation for evaluating .Machine learning methods also could be used to train more directly from simulation data for sampled colloid trajectories . The training would select an ML model over some class parameterized by , such as the weights and biases of a Deep Neural Network. For instance, this could be done by Maximum Likelihood Estimation (MLE) from the trajectory data by optimizing the objective
| (2) |
The
denotes the probability density when
is used in equation 1 for observing the configurations . To obtain tractable and robust training algorithms further approximations and regularizations may be required to the MLE problem in equation 2. This could include using variational inference approaches, further restrictions on the model architectures, or introducing priors [17, 16, 5]. Combining such approximations with further regularizations could help facilitate learning possible symmetries and other features shaping the learned model .The MLMOD package provides a way for transferring such learned models from the data into components of practical simulations in LAMMPS. We discussed here one example of a basic data-driven modeling approach. The MLMOD package can be used more generally and support more broad classes of models for incorporating machine learning results into simulation components. Components can include the dynamics, interactions, and for computing quantities of interest as system state information or characterizing system behaviors. The initial prototype implementation we present in this whitepaper supports the basic mobility modeling framework as a proof-of-concept, with longer-term aims to support more general classes of reduced dynamics and interactions in future releases.
Structure of the Package Components
The package is organized as a module within LAMMPS that is called each time-step and has the potential to serve multiple functions in simulations. This includes (i) serving as a time-step integrator updating the configuration of the system based on a specified learned model, (ii) evaluating interactions between system components to compute energy and forces, and (iii) compute quantities of interest (QoI) that can be used as state information during simulations or computing statistics. The package is controlled by external XML files that specify the mode of operation and source for pre-trained models and other information, see the schematic in Figure 2.
.
The MLMOD Package is incorporated into a simulation by either using the LAMMPS scripting language or the python interface. This is done using the ”fix” command in LAMMPS, with this terminology historically motivated by algorithms for ”fixing” molecular bonds as rigid each time-step. For our package the command to set up the triggers for our algorithms is fix 1 mlmod all filename.mlmod_params. This specifies the tag ”1” for this fix, particle groups controlled by the package as ”all”, and the XML file of parameters. The XML file filename.mlmod_params specifies the MLMOD simulation mode and where to find the associated exported ML models. An example and more details are discussed below in the section on package usage. The MLMOD Package can evaluate machine learning models using frameworks such as C++ PyTorch API. While this would also allow for the possibility of doing on-the-fly learning, we anticipate the most common use scenarios would be to train in advance to learn models and then incorporate them into simulations through evaluation.
A common scenario we anticipate is for a data-driven model to be obtained from a machine learning framework such as PyTorch by training from trajectory data from detailed high fidelity simulations. Once the model is trained, it can be exported to a portable format such as Torch. The MLMOD package would import these pre-trained models from Torch files such as ”trained_model.pt”. This allows for these models to then be invoked by MLMOD to provide elements for (i) performing time-step integration to model dynamics, (ii) computing interactions between system components, and (iii) computing quantities of interest (QoI) for further computations or as statistics. This provides a modular and general way for data-driven models obtained from training with machine learning methods to be used to govern LAMMPS simulations.
Example Usage of the Package
We give one basic example usage of the package in the case for modeling colloids using a mobility tensor . The MLMOD package is invoked by using the ”fix” command in LAMMPS, with this terminology historically motivated by algorithms for ”fixing” molecular bonds as rigid each time-step. To set up the triggers for the MLMOD package during LAMMPS simulations a typical command would look like
fix m1 c_group mlmod model.mlmod_params
The m1 gives the tag for the fix, c_group specifies the label for the group of particles controlled by this instance of the MLMOD package, the mlmod specifies to use the MLMOD package, and model.mlmod_params gives the XML file with parameters controlling the mode to run MLMOD package and associated exported ML models.
Multiple instances of MLMOD package are permitted to control different groups of particles. The package is designed with modularity so a mode is first defined in a parameter file and then different sets of algorithms and parameters are used based on that information. For the mobility example discussed in the introduction, a basic prototype implementation is given by the MLMOD simulation mode dX_MF_ML1. For this modeling mode, a typical parameter file would look like
<?xml version="1.0" encoding="UTF-8"?> <MLMOD> <model_data type="dX_MF_ML1"> <M_ii_filename value="M_ii_torch.pt"/> <M_ij_filename value="M_ij_torch.pt"/> </model_data> </MLMOD>
This specifies for an assumed mobility tensor of pairwise interactions the models for the self-mobility responses and the pairwise mobility response . For example, a hydrodynamic model for interactions when the two colloids of radius are not too close together is to use the Oseen Tensors and . The is the fluid viscosity, with give the particle separation. The responses are with and summation notation.
The dX_MF_ML1 mode allows for this type of model and replacing the interactions with ML models learned from simulation data. Related modes can also be implemented to extend models to capture more complicated interactions or near-field effects. For example, to allow for localized many-body interactions with ML models giving contributions to mobility . In this way MLMOD can be used for hybrid modeling combining ML models with more traditional modeling approaches within a unified framework.
The models used by MLMOD in principle can be of any exported format from machine learning frameworks. Currently, the implementation uses PyTorch and the export format based on torch script with .pt files. This allows for a variety of models to be used ranging from those based on Deep Neural Networks, Kernel Regression Models, and others. We describe here our early work on a prototype implementation, so some aspects of the interface may change in future releases. For examples, updates, and additional information please check the MLMOD package website at http://software.atzberger.org/.
Conclusion
The package provides capabilities in LAMMPS for incorporating into simulations data-driven models for dynamics and interactions obtained from training with machine learning methods. The package supports representations using Neural Networks, Gaussian Process Regression, Kernel Models, and other classes. Discussed here is our early prototype implementation of the MLMOD package. Please check the MLMOD website for updates and future releases at http://software.atzberger.org/.
Acknowledgments
Authors research supported by grants DOE Grant ASCR PHILMS DE-SC0019246 and NSF Grant DMS-1616353. Authors also acknowledge UCSB Center for Scientific Computing NSF MRSEC (DMR1121053) and UCSB MRL NSF CNS-1725797. P.J.A. would also like to acknowledge a hardware grant from Nvidia.
References
- Atzberger [2013] Atzberger, P. J. 2013. Incorporating shear into stochastic Eulerian-Lagrangian methods for rheological studies of complex fluids and soft materials. Physica D: Nonlinear Phenomena 265: 57–70. ISSN 0167-2789. doi:https://doi.org/10.1016/j.physd.2013.09.002. URL https://www.sciencedirect.com/science/article/pii/S016727891300256X.
- Atzberger [2018] Atzberger, P. J. 2018. Importance of the Mathematical Foundations of Machine Learning Methods for Scientific and Engineering Applications. SciML2018 Workshop, position paper URL https://arxiv.org/abs/1808.02213.
- Bauer, Thorpe, and Brunet [2015] Bauer, P.; Thorpe, A.; and Brunet, G. 2015. The quiet revolution of numerical weather prediction. Nature 525(7567): 47–55. ISSN 1476-4687. URL https://doi.org/10.1038/nature14956.
- Bird [1987] Bird, R.B., C. C. F. A. R. C. H. O. 1987. Dynamics of Polymeric Liquids : Volume II Kinetic Theory. Wiley-Interscience.
- Blei, Kucukelbir, and McAuliffe [2017] Blei, D. M.; Kucukelbir, A.; and McAuliffe, J. D. 2017. Variational Inference: A Review for Statisticians. Journal of the American Statistical Association 112(518): 859–877. doi:10.1080/01621459.2017.1285773. URL https://doi.org/10.1080/01621459.2017.1285773.
- Brooks et al. [1983] Brooks, B. R.; Bruccoleri, R. E.; Olafson, B. D.; States, D. J.; Swaminathan, S. a.; and Karplus, M. 1983. CHARMM: a program for macromolecular energy, minimization, and dynamics calculations. Journal of computational chemistry 4(2): 187–217.
- Brunton, Proctor, and Kutz [2016] Brunton, S. L.; Proctor, J. L.; and Kutz, J. N. 2016. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences 113(15): 3932–3937. ISSN 0027-8424. doi:10.1073/pnas.1517384113. URL https://www.pnas.org/content/113/15/3932.
- Derjaguin [1941] Derjaguin, B.; Landau, L. 1941. Theory of the stability of strongly charged lyophobic sols and of the adhesion of strongly charged particles in solutions of electrolytes. Acta Physico Chemica URSS 633(14).
- Doi [2013] Doi, M. 2013. Soft matter physics. Oxford University Press.
- Goodfellow, Bengio, and Courville [2016] Goodfellow, I.; Bengio, Y.; and Courville, A. 2016. Deep Learning. The MIT Press. ISBN 0262035618. URL https://www.deeplearningbook.org/.
- Hastie, Tibshirani, and Friedman [2001] Hastie, T.; Tibshirani, R.; and Friedman, J. 2001. Elements of Statistical Learning. Springer Series in Statistics. New York, NY, USA: Springer New York Inc.
- Hinton and Salakhutdinov [2006] Hinton, G.; and Salakhutdinov, R. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313(5786): 504–507. doi:10.1126/science.1127647. URL https://www.cs.toronto.edu/~hinton/science.pdf.
- Jones, Jones, and R Jones [2002] Jones, R.; Jones, R.; and R Jones, P. 2002. Soft Condensed Matter. Oxford Master Series in Physics. OUP Oxford. ISBN 9780198505891. URL https://books.google.com/books?id=Hl_HBPUvoNsC.
- Karplus and McCammon [2002] Karplus, M.; and McCammon, J. A. 2002. Molecular dynamics simulations of biomolecules. Nature Structural Biology 9(9): 646–652. ISSN 1545-9985. URL https://doi.org/10.1038/nsb0902-646.
- Kimura [2009] Kimura, Y. 2009. Microrheology of Soft Matter. J. Phys. Soc. Jpn. 78(4): 8–. ISSN 0031-9015.
- Kingma and Welling [2014] Kingma, D. P.; and Welling, M. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings. URL http://arxiv.org/abs/1312.6114.
- Lopez and Atzberger [2020] Lopez, R.; and Atzberger, P. J. 2020. Variational Autoencoders for Learning Nonlinear Dynamics of Physical Systems URL https://arxiv.org/abs/2012.03448.
- Lubensky [1997] Lubensky, T. C. 1997. Soft condensed matter physics. Solid State Communications 102(2-3): 187–197–. ISSN 0038-1098.
- Lusk and Mattsson [2011] Lusk, M. T.; and Mattsson, A. E. 2011. High-performance computing for materials design to advance energy science. MRS Bulletin 36(3): 169–174. ISSN 1938-1425. URL https://doi.org/10.1557/mrs.2011.30.
- McCammon and Harvey [1988] McCammon, J.; and Harvey, S. 1988. Dynamics of Proteins and Nucleic Acids. Cambridge University Press. ISBN 9780521356527. URL https://books.google.com/books?id=OXDS3Oq2QYIC.
- Murr [2016] Murr, L. E. 2016. Computer Simulations in Materials Science and Engineering, 1–15. Cham: Springer International Publishing. ISBN 978-3-319-01905-5. doi:10.1007/978-3-319-01905-5˙60-2. URL https://doi.org/10.1007/978-3-319-01905-5˙60-2.
- Pan [2021] Pan, J. 2021. Scaling up system size in materials simulation. Nature Computational Science 1(2): 95–95. ISSN 2662-8457. URL https://doi.org/10.1038/s43588-021-00034-x.
- Plimpton [1995] Plimpton, S. 1995. Fast Parallel Algorithms for Short-Range Molecular Dynamics. Journal of Computational Physics 117(1): 1–19. ISSN 0021-9991. URL http://www.sciencedirect.com/science/article/pii/S002199918571039X.
- Rasmussen [2004] Rasmussen, C. E. 2004. Gaussian Processes in Machine Learning, 63–71. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-540-28650-9. doi:10.1007/978-3-540-28650-9˙4. URL https://doi.org/10.1007/978-3-540-28650-9˙4.
- Richardson [2007] Richardson, L. F. 2007. Weather prediction by numerical process. Cambridge university press.
- Sanbonmatsu and Tung [2007] Sanbonmatsu, K.; and Tung, C.-S. 2007. High performance computing in biology: multimillion atom simulations of nanoscale systems. Journal of structural biology 157(3): 470–480.
- Schmidt and Lipson [2009] Schmidt, M.; and Lipson, H. 2009. Distilling Free-Form Natural Laws from Experimental Data 324: 81–85. ISSN 0036-8075. doi:10.1126/science.1165893.
-
Scholkopf and Smola [2001]
Scholkopf, B.; and Smola, A. J. 2001.
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
. Cambridge, MA, USA: MIT Press. ISBN 0262194759. - Sidhu, Frischknecht, and Atzberger [2018] Sidhu, I.; Frischknecht, A. L.; and Atzberger, P. J. 2018. Electrostatics of Nanoparticle-Wall Interactions within Nanochannels: Role of Double-Layer Structure and Ion-Ion Correlations. ACS Omega 3(9): 11340–11353. ISSN 2470-1343. doi:10.1021/acsomega.8b01393. URL https://doi.org/10.1021/acsomega.8b01393.
- van der Giessen et al. [2020] van der Giessen, E.; Schultz, P. A.; Bertin, N.; Bulatov, V. V.; Cai, W.; Csányi, G.; Foiles, S. M.; Geers, M. G. D.; González, C.; Hütter, M.; Kim, W. K.; Kochmann, D. M.; LLorca, J.; Mattsson, A. E.; Rottler, J.; Shluger, A.; Sills, R. B.; Steinbach, I.; Strachan, A.; and Tadmor, E. B. 2020. Roadmap on multiscale materials modeling. Modelling and Simulation in Materials Science and Engineering 28(4): 043001. doi:10.1088/1361-651x/ab7150. URL https://doi.org/10.1088/1361-651x/ab7150.
- Washington, Buja, and Craig [2009] Washington, W. M.; Buja, L.; and Craig, A. 2009. The computational future for climate and Earth system models: on the path to petaflop and beyond. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 367(1890): 833–846. doi:10.1098/rsta.2008.0219. URL https://royalsocietypublishing.org/doi/abs/10.1098/rsta.2008.0219.
Comments
There are no comments yet.