Vid2Param: Online system identification from video for robotics applications

by   Martin Asenov, et al.

Robots performing tasks in dynamic environments would benefit greatly from understanding the underlying environment motion, in order to make future predictions and to synthesize effective control policies that use this inductive bias. Online system identification is therefore a fundamental requirement for robust autonomous agents. When the dynamics involves multiple modes (due to contacts or interactions between objects), and when system identification must proceed directly from a rich sensory stream such as video, then traditional methods for system identification may not be well suited. We propose an approach wherein fast parameter estimation with a model can be seamlessly combined with a recurrent variational autoencoder. Our Physics-based recurrent variational autoencoder model includes an additional loss that enforces conformity with the structure of a physically based dynamics model. This enables the resulting model to encode parameters such as position, velocity, restitution, air drag and other physical properties of the system. The model can be trained entirely in simulation, in an end-to-end manner with domain randomization, to perform online system identification, and probabilistic forward predictions of parameters of interest. We benchmark against existing system identification methods and demonstrate that Vid2Param outperforms the baselines in terms of speed and accuracy of identification, and also provides uncertainty quantification in the form of a distribution over future trajectories. Furthermore, we illustrate the utility of this in physical experiments wherein a PR2 robot with velocity constrained arm must intercept a bouncing ball, by estimating the physical parameters of this ball directly from the video trace after the ball is released.


page 4

page 7

page 8


Physics-enhanced Gaussian Process Variational Autoencoder

Variational autoencoders allow to learn a lower-dimensional latent space...

Autonomous Golf Putting with Data-Driven and Physics-Based Methods

We are developing a self-learning mechatronic golf robot using combined ...

Physics-as-Inverse-Graphics: Joint Unsupervised Learning of Objects and Physics from Video

We aim to perform unsupervised discovery of objects and their states suc...

Semiparametrical Gaussian Processes Learning of Forward Dynamical Models for Navigating in a Circular Maze

This paper presents a problem of model learning for the purpose of learn...

STReSSD: Sim-To-Real from Sound for Stochastic Dynamics

Sound is an information-rich medium that captures dynamic physical event...

BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators

We introduce BayesSim, a framework for robotics simulations allowing a f...

Joint Parameter Discovery and Generative Modeling of Dynamic Systems

Given an unknown dynamic system such as a coupled harmonic oscillator wi...

Please sign up or login with your details

Forgot password? Click here to reset