Virtual-to-Real: Learning to Control in Visual Semantic Segmentation

02/01/2018
by   Zhang-Wei Hong, et al.
0

Collecting training data from the physical world is usually time-consuming and even dangerous for fragile robots, and thus, recent advances in robot learning advocate the use of simulators as the training platform. Unfortunately, the reality gap between synthetic and real visual data prohibits direct migration of the models trained in virtual worlds to the real world. This paper proposes a modular architecture for tackling the virtual-to-real problem. The proposed architecture separates the learning model into a perception module and a control policy module, and uses semantic image segmentation as the meta representation for relating these two modules. The perception module translates the perceived RGB image to semantic image segmentation. The control policy module is implemented as a deep reinforcement learning agent, which performs actions based on the translated image segmentation. Our architecture is evaluated in an obstacle avoidance task and a target following task. Experimental results show that our architecture significantly outperforms all of the baseline methods in both virtual and real environments, and demonstrates a faster learning curve than them. We also present a detailed analysis for a variety of variant configurations, and validate the transferability of our modular architecture.

READ FULL TEXT
research
01/13/2018

Autonomous Driving in Reality with Reinforcement Learning and Image Translation

Supervised learning is widely used in training autonomous driving vehicl...
research
07/20/2022

DeepIPC: Deeply Integrated Perception and Control for Mobile Robot in Real Environments

We propose DeepIPC, an end-to-end multi-task model that handles both per...
research
10/31/2022

Learning Modular Robot Visual-motor Locomotion Policies

Control policy learning for modular robot locomotion has previously been...
research
09/06/2023

Multi-log grasping using reinforcement learning and virtual visual servoing

We explore multi-log grasping using reinforcement learning and virtual v...
research
02/25/2020

Learning Machines from Simulation to Real World

Learning Machines is developing a flexible, cross-industry, advanced ana...
research
05/09/2022

Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes

This paper proposes a learning-based visual peg-in-hole that enables tra...
research
04/16/2022

Learning to Fill the Seam by Vision: Sub-millimeter Peg-in-hole on Unseen Shapes in Real World

In the peg insertion task, human pays attention to the seam between the ...

Please sign up or login with your details

Forgot password? Click here to reset