I Introduction and Related Work
Simulators are crucial tools for planning and control algorithms to tackle difficult real world robotics problems. In many cases, however, such models diverge from reality in important ways, leading to algorithms that work well in simulation and fail in reality. Closing the sim2real gap has gained significant interest, and various dynamics modeling approaches have been proposed (Figure 2 left).
Various methods learn system dynamics from time series data of a real system. Such “intuitive physics” models often use deep graph neural networks to discover constraints between particles or bodies [5, 34, 14, 30, 28, 7, 24, 31]. We propose a generalpurpose hybrid simulation approach that combines analytical models of dynamical systems with datadriven residual models that learn parts of the dynamics unaccounted for by the analytical simulation models.
Originating from traditional physics engines [8, 33, 23], differentiable simulators have been introduced that leverage automatic, symbolic or implicit differentiation to calculate parameter gradients through the analytical physics models for rigidbody dynamics [11, 6, 10, 22, 17, 16], light propagation [29, 15], and other phenomena [18, 25].
Residual physics models [35, 2, 20, 12] augment physics engines with learned models to reduce the sim2real gap. Most of them introduce residual learning applied to the output of the physics engine, while we propose a more finegrained approach, similar to Hwangbo et al. [20], where only at some parts in the simulator datadriven models are introduced. While in [20]
the network for actuator dynamics is trained through supervised learning, our endtoend differentiable model allows backpropagation of gradients from highlevel states to any part of the computation graph, including neural network weights, so that these parameters can be optimized efficiently, for example from endeffector trajectories.
Ii Approach
We propose a technique for hybrid simulation that leverages differentiable physics models and neural networks to allow for efficient system identification, design optimization, and gradientbased trajectory planning. By enabling any part of the simulation to be replaced or augmented by neural networks, we can learn unmodeled effects from data. Through template metaprogramming, our opensource C++ implementation
^{1}^{1}1https://github.com/googleresearch/tinydifferentiablesimulator allows any variable participating in the simulation to be augmented by neural networks that accept input connections from any other variable. In the simulation code, such neural scalars (Figure 2 right) are assigned a unique name, so that in a separate experiment code a “neural blueprint” is defined that declares the neural network architectures and sets the network weights. We compute gradients of the weights and analytical simulation parameters using the multidimensional dual number implementation from Ceres [1] and have support for many other automatic differentiation libraries.Analytical  Datadriven  End2end  Hybrid  

Physics engine [8, 33, 23]  ✓  
Residual physics [20, 2, 35, 12]  ✓  ✓  ✓  
Learned physics [32, 5, 24, 21]  ✓  ✓  
Differentiable sim. [11, 18, 6, 10]  ✓  ✓  
Ours  ✓  ✓  ✓  ✓ 
Iii System Identification
Given the state trajectory from the target system, we optimize the following loss for system identification:
(1) 
where is the discrete dynamics function mapping from the previous simulation state to the current state
which is implemented by our physics engine given the parameter vector
that consists of the parameters for the analytical model, plus the parameters that correspond to the weights of the neural networks in the simulation. To ensure the residual dynamics learned by the neural networks are minimal, we regularize the network weights by factor which penalizes large state contributions.Iv Overcoming Local Optima
We solve the nonlinear least squares problem from Equation 1
using the LevenbergMarquardt algorithm (LMA). Such a gradientbased optimization method quickly finds local optima, but due to the highly nonconvex loss landscapes commonly encountered in system identification problems for nonlinear dynamics, the resulting parameter estimates often exhibit a poor fit to the real world data. To escape such poor local minima, we adapt a random search strategy,
parallel basin hopping (PBH) [27], that, in our instantiation, runs multiple LMA solver and simulation instances in parallel while continuously randomizing the initial parameters from which the local solvers are restarted after convergence criteria, time limits, or maximum iteration counts are met.V Results
We present preliminary results for sim2sim transfer to match the hybrid dynamics model to richer analytical simulations. Additionally, we demonstrate our system identification approach on a realworld dataset.
In our first experiment, we transfer rigidbody contact dynamics simulated using a velocitylevel contact model formulated as a linear complementarity problem [3]. Our hybrid simulator uses a pointbased nonlinearspring contact model where the normal force is solved analytically through the HuntCrossley model [19] and the friction force is learned by a neural network that receives the relative velocities, contact normal force and penetration depth as input. Before optimizing the analytical and neural model parameters, the trajectories of a cube thrown horizontally on the ground differ dramatically. After system identification using PBH applied on Equation 1 given trajectories of positions and velocities from the target system, the gap is significantly reduced (Figure 1 left).
In the next experiment, we apply our approach to a realworld dynamical system. Given joint position trajectories from the doublependulum dataset provided by Asseman et al. [4], we optimize inertia, masses, and link lengths of our simulated model and achieve a minimal sim2real gap (Figure 1 right).
Vi Conclusion
We have demonstrated a simulation architecture that allows us to insert neural networks at any place in a differentiable physics engine to augment analytical models with the ability to learn dynamical effects from data. In our preliminary experiments, efficient gradientbased optimizers quickly converge to simulations that closely follow the observed trajectories from the target systems, while poor local minima are overcome through a random search strategy.
Future research is directed towards more automated ways to identify where such extra degrees of freedom are needed to close the sim2real gap given a few trajectories from the real system. Our loss function in
Equation 1 regularizes the contributions of the neural networks to the overall system dynamics. Nonetheless, this approach does not prevent violating basic laws of physics, such as energy and momentum preservation. Hamiltonian [13] and (Deep) Lagrangian neural networks [26, 9] explicitly constrain the solution space to remain consistent with such principles but need to be further investigated in the context of residual models in hybrid simulators.References
 [1] Ceres solver. Note: http://ceressolver.org Cited by: §II.
 [2] (2018) Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Cited by: Fig. 2, §I, Table I.
 [3] (1997) Formulating dynamic multirigidbody contact problems with friction as solvable linear complementarity problems. Nonlinear Dynamics 14 (3), pp. 231–247. Cited by: §V.
 [4] (2018) Learning beyond simulated physics. In Modeling and Decisionmaking in the Spatiotemporal Domain Workshop, External Links: Link Cited by: Fig. 1, §V.
 [5] (2016) Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems, pp. 4502–4510. Cited by: §I, Table I.
 [6] (2018) Analytical derivatives of rigid body dynamics algorithms. In Robotics: Science and Systems, Cited by: §I, Table I.
 [7] (2018) . In Advances in Neural Information Processing Systems, pp. 6571–6583. Cited by: §I.
 [8] (2013) Bullet physics library. Open source: bulletphysics.org 15 (49), pp. 5. Cited by: §I, Table I.
 [9] (2020) Lagrangian neural networks. External Links: 2003.04630 Cited by: §VI.
 [10] (2018) Endtoend differentiable physics for learning and control. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.), pp. 7178–7189. External Links: Link Cited by: §I, Table I.
 [11] (2017) Automatic differentiation of rigid body dynamics for optimal control and estimation. Advanced Robotics 31 (22), pp. 1225–1237. External Links: Document Cited by: §I, Table I.

[12]
(201829–31 Oct)
Simtoreal transfer with neuralaugmented robot simulation.
In Proceedings of The 2nd Conference on Robot Learning, A. Billard, A. Dragan, J. Peters, and J. Morimoto (Eds.),
Proceedings of Machine Learning Research
, Vol. 87, , pp. 817–828. External Links: Link Cited by: §I, Table I.  [13] (2019) Hamiltonian neural networks. arXiv preprint arXiv:1906.01563. Cited by: §VI.
 [14] (201907) Learning to predict the cosmological structure formation. Proceedings of the National Academy of Sciences of the United States of America 116 (28), pp. 13825–13832 (en). Cited by: §I.
 [15] (2020) Physicsbased simulation of continuouswave LIDAR for localization, calibration and tracking. In International Conference on Robotics and Automation (ICRA), Cited by: §I.
 [16] (2019) Real2Sim transfer using differentiable physics. R:SS Workshop on Closing the Reality Gap in Sim2real Transfer for Robotic Manipulation. Cited by: §I.
 [17] (2019) Interactive differentiable simulation. CoRR abs/1905.10706. External Links: Link, 1905.10706 Cited by: §I.
 [18] (2020) DiffTaichi: differentiable programming for physical simulation. ICLR. Cited by: §I, Table I.
 [19] (1975) Coefficient of restitution interpreted as damping in vibroimpact. Cited by: §V.
 [20] (2019) Learning agile and dynamic motor skills for legged robots. Science Robotics 4 (26), pp. eaau5872. Cited by: §I, Table I.
 [21] (2018) Dataaugmented contact model for rigid body simulation. CoRR abs/1803.04019. External Links: Link, 1803.04019 Cited by: Table I.
 [22] (201905) Julia for robotics: simulation and realtime control in a highlevel programming language. In International Conference on Robotics and Automation, pp. . Cited by: §I.
 [23] (2018) DART: dynamic animation and robotics toolkit. Journal of Open Source Software 3 (22), pp. 500. External Links: Document, Link Cited by: §I, Table I.
 [24] (2019) Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, External Links: Link Cited by: §I, Table I.
 [25] (2019) Differentiable cloth simulation for inverse problems. In Advances in Neural Information Processing Systems, pp. 771–780. Cited by: §I.

[26]
(2019)
Deep lagrangian networks: using physics as model prior for deep learning
. arXiv preprint arXiv:1907.04490. Cited by: §VI.  [27] (2018) Parallel monotonic basin hopping for low thrust trajectory optimization. In 2018 Space Flight Mechanics Meeting, pp. 1452. Cited by: §IV.
 [28] (201806) Flexible neural representation for physics prediction. Advances in Neural Information Processing Systems. Cited by: §I.
 [29] (201911) Mitsuba 2: a retargetable forward and inverse renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38 (6). External Links: Document Cited by: §I.
 [30] (2018) Deep learning of turbulent scalar mixing. Technical report Cited by: §I.
 [31] (2020) Learning to simulate complex physics with graph networks. External Links: 2002.09405 Cited by: §I.
 [32] (2020) Learning to simulate complex physics with graph networks. arXiv preprint arXiv:2002.09405. Cited by: Table I.
 [33] (2012) Mujoco: a physics engine for modelbased control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033. Cited by: §I, Table I.
 [34] (201906) DensePhysNet: learning dense physical object representations via multistep dynamic interactions. Robotics: Science and Systems. Cited by: §I.
 [35] (2019) TossingBot: learning to throw arbitrary objects with residual physics. Cited by: §I, Table I.