DeepAI AI Chat
Log In Sign Up

Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

10/25/2022
by   John So, et al.
0

Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.

READ FULL TEXT

page 1

page 4

page 5

page 6

page 7

10/22/2021

ModEL: A Modularized End-to-end Reinforcement Learning Framework for Autonomous Driving

Heated debates continue over the best autonomous driving framework. The ...
11/29/2019

Simulation-based reinforcement learning for real-world autonomous driving

We use synthetic data and a reinforcement learning algorithm to train a ...
11/12/2021

DriverGym: Democratising Reinforcement Learning for Autonomous Driving

Despite promising progress in reinforcement learning (RL), developing al...
10/12/2017

Arguing Machines: Perception-Control System Redundancy and Edge Case Discovery in Real-World Autonomous Driving

Safe autonomous driving may be one of the most difficult engineering cha...
05/02/2021

A LiDAR Assisted Control Module with High Precision in Parking Scenarios for Autonomous Driving Vehicle

Autonomous driving has been quite promising in recent years. The public ...
11/14/2022

NeurIPS 2022 Competition: Driving SMARTS

Driving SMARTS is a regular competition designed to tackle problems caus...
10/02/2021

Cycle-Consistent World Models for Domain Independent Latent Imagination

End-to-end autonomous driving seeks to solve the perception, decision, a...