NeuralSim: Augmenting Differentiable Simulators with Neural Networks

by   Eric Heiden, et al.

Differentiable simulators provide an avenue for closing the sim-to-real gap by enabling the use of efficient, gradient-based optimization algorithms to find the simulation parameters that best fit the observed sensor readings. Nonetheless, these analytical models can only predict the dynamical behavior of systems for which they have been designed. In this work, we study the augmentation of a novel differentiable rigid-body physics engine via neural networks that is able to learn nonlinear relationships between dynamic quantities and can thus learn effects not accounted for in traditional simulators.Such augmentations require less data to train and generalize better compared to entirely data-driven models. Through extensive experiments, we demonstrate the ability of our hybrid simulator to learn complex dynamics involving frictional contacts from real data, as well as match known models of viscous friction, and present an approach for automatically discovering useful augmentations. We show that, besides benefiting dynamics modeling, inserting neural networks can accelerate model-based control architectures. We observe a ten-fold speed-up when replacing the QP solver inside a model-predictive gait controller for quadruped robots with a neural network, allowing us to significantly improve control delays as we demonstrate in real-hardware experiments. We publish code, additional results and videos from our experiments on our project webpage at


page 1

page 3

page 4

page 5


Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gap

We present a differentiable simulation architecture for articulated rigi...

Differentiable Hybrid Traffic Simulation

We introduce a novel differentiable hybrid traffic simulator, which simu...

Automatic Differentiation and Continuous Sensitivity Analysis of Rigid Body Dynamics

A key ingredient to achieving intelligent behavior is physical understan...

Parameter Identification and Motion Control for Articulated Rigid Body Robots Using Differentiable Position-based Dynamics

Simulation modeling of robots, objects, and environments is the backbone...

Combining Physical Simulators and Object-Based Networks for Control

Physics engines play an important role in robot planning and control; ho...

Interactive Differentiable Simulation

Intelligent agents need a physical understanding of the world to predict...

DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles

Autonomous vehicle (AV) stacks are typically built in a modular fashion,...

Please sign up or login with your details

Forgot password? Click here to reset