Close the Visual Domain Gap by Physics-Grounded Active Stereovision Depth Sensor Simulation

by   Xiaoshuai Zhang, et al.
Tsinghua University
University of California, San Diego

In this paper, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by the underlying mechanism of the sensors, we designed a fully physics-grounded simulation pipeline, which includes material acquisition, ray tracing based infrared (IR) image rendering, IR noise simulation, and depth estimation. The pipeline is able to generate depth maps with material-dependent error patterns similar to a real depth sensor. We conduct extensive experiments to show that perception algorithms and reinforcement learning policies trained in our simulation platform could transfer well to real world test cases without any fine-tuning. Furthermore, due to the high degree of realism of this simulation, our depth sensor simulator can be used as a convenient testbed to evaluate the algorithm performance in the real world, which will largely reduce the human effort in developing robotic algorithms. The entire pipeline has been integrated into the SAPIEN simulator and is open-sourced to promote the research of vision and robotics communities.


page 2

page 5

page 9

page 10

page 11

page 13

page 14

page 16


Simulation of Vision-based Tactile Sensors using Physics based Rendering

Tactile sensing has seen a rapid adoption with the advent of vision-base...

ActiveZero: Mixed Domain Learning for Active Stereovision with Zero Annotation

Traditional depth sensors generate accurate real world depth estimates t...

Reinforced Grounded Action Transformation for Sim-to-Real Transfer

Robots can learn to do complex tasks in simulation, but often, learned b...

OmniLRS: A Photorealistic Simulator for Lunar Robotics

Developing algorithms for extra-terrestrial robotic exploration has alwa...

Monocular Depth Estimation for Soft Visuotactile Sensors

Fluid-filled soft visuotactile sensors such as the Soft-bubbles alleviat...

Aberration-Aware Depth-from-Focus

Computer vision methods for depth estimation usually use simple camera m...

Please sign up or login with your details

Forgot password? Click here to reset