Learning a generative model for robot control using visual feedback

by   Nishad Gothoskar, et al.

We introduce a novel formulation for incorporating visual feedback in controlling robots. We define a generative model from actions to image observations of features on the end-effector. Inference in the model allows us to infer the robot state corresponding to target locations of the features. This, in turn, guides motion of the robot and allows for matching the target locations of the features in significantly fewer steps than state-of-the-art visual servoing methods. The training procedure for our model enables effective learning of the kinematics, feature structure, and camera parameters, simultaneously. This can be done with no prior information about the robot, structure, and cameras that observe it. Learning is done sample-efficiently and shows strong generalization to test data. Since our formulation is modular, we can modify components of our setup, like cameras and objects, and relearn them quickly online. Our method can handle noise in the observed state and noise in the controllers that we interact with. We demonstrate the effectiveness of our method by executing grasping and tight-fit insertions on robots with inaccurate controllers.


page 14

page 15


DURableVS: Data-efficient Unsupervised Recalibrating Visual Servoing via online learning in a structured generative model

Visual servoing enables robotic systems to perform accurate closed-loop ...

Learning Manipulation States and Actions for Efficient Non-prehensile Rearrangement Planning

This paper addresses non-prehensile rearrangement planning problems wher...

Predicting Target Feature Configuration of Non-stationary Objects for Grasping with Image-Based Visual Servoing

In this paper we consider the problem of the final approach stage of clo...

Morphology-Agnostic Visual Robotic Control

Existing approaches for visuomotor robotic control typically require cha...

Detecting Features of Tools, Objects, and Actions from Effects in a Robot using Deep Learning

We propose a tool-use model that can detect the features of tools, targe...

TossingBot: Learning to Throw Arbitrary Objects with Residual Physics

We investigate whether a robot arm can learn to pick and throw arbitrary...

Please sign up or login with your details

Forgot password? Click here to reset