DeepAI AI Chat
Log In Sign Up

Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

by   John So, et al.

Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.


page 1

page 4

page 5

page 6

page 7


ModEL: A Modularized End-to-end Reinforcement Learning Framework for Autonomous Driving

Heated debates continue over the best autonomous driving framework. The ...

Simulation-based reinforcement learning for real-world autonomous driving

We use synthetic data and a reinforcement learning algorithm to train a ...

DriverGym: Democratising Reinforcement Learning for Autonomous Driving

Despite promising progress in reinforcement learning (RL), developing al...

Arguing Machines: Perception-Control System Redundancy and Edge Case Discovery in Real-World Autonomous Driving

Safe autonomous driving may be one of the most difficult engineering cha...

A LiDAR Assisted Control Module with High Precision in Parking Scenarios for Autonomous Driving Vehicle

Autonomous driving has been quite promising in recent years. The public ...

NeurIPS 2022 Competition: Driving SMARTS

Driving SMARTS is a regular competition designed to tackle problems caus...

Cycle-Consistent World Models for Domain Independent Latent Imagination

End-to-end autonomous driving seeks to solve the perception, decision, a...