Log In Sign Up

Cycle-Consistent World Models for Domain Independent Latent Imagination

by   Sidney Bender, et al.

End-to-end autonomous driving seeks to solve the perception, decision, and control problems in an integrated way, which can be easier to generalize at scale and be more adapting to new scenarios. However, high costs and risks make it very hard to train autonomous cars in the real world. Simulations can therefore be a powerful tool to enable training. Due to slightly different observations, agents trained and evaluated solely in simulation often perform well there but have difficulties in real-world environments. To tackle this problem, we propose a novel model-based reinforcement learning approach called Cycleconsistent World Models. Contrary to related approaches, our model can embed two modalities in a shared latent space and thereby learn from samples in one modality (e.g., simulated data) and be used for inference in different domain (e.g., real-world data). Our experiments using different modalities in the CARLA simulator showed that this enables CCWM to outperform state-of-the-art domain adaptation approaches. Furthermore, we show that CCWM can decode a given latent representation into semantically coherent observations in both modalities.


page 3

page 6

page 8


From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning

Deep Reinforcement Learning has proved to be able to solve many control ...

Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning

Unlike popular modularized framework, end-to-end autonomous driving seek...

DR2L: Surfacing Corner Cases to Robustify Autonomous Driving via Domain Randomization Reinforcement Learning

How to explore corner cases as efficiently and thoroughly as possible ha...

Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation

Despite the recent success of deep reinforcement learning (RL), domain a...

Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data

Autonomous driving is complex, requiring sophisticated 3D scene understa...

End-to-end Autonomous Driving Perception with Sequential Latent Representation Learning

Current autonomous driving systems are composed of a perception system a...