Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space

by   Pin Wang, et al.

Ramp merging is a critical maneuver for road safety and traffic efficiency. Most of the current automated driving systems developed by multiple automobile manufacturers and suppliers are typically limited to restricted access freeways only. Extending the automated mode to ramp merging zones presents substantial challenges. One is that the automated vehicle needs to incorporate a future objective (e.g. a successful and smooth merge) and optimize a long-term reward that is impacted by subsequent actions when executing the current action. Furthermore, the merging process involves interaction between the merging vehicle and its surrounding vehicles whose behavior may be cooperative or adversarial, leading to distinct merging countermeasures that are crucial to successfully complete the merge. In place of the conventional rule-based approaches, we propose to apply reinforcement learning algorithm on the automated vehicle agent to find an optimal driving policy by maximizing the long-term reward in an interactive driving environment. Most importantly, in contrast to most reinforcement learning applications in which the action space is resolved as discrete, our approach treats the action space as well as the state space as continuous without incurring additional computational costs. Our unique contribution is the design of the Q-function approximation whose format is structured as a quadratic function, by which simple but effective neural networks are used to estimate its coefficients. The results obtained through the implementation of our training platform demonstrate that the vehicle agent is able to learn a safe, smooth and timely merging policy, indicating the effectiveness and practicality of our approach.


page 1

page 2

page 3

page 4


Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge

Multiple automakers have in development or in production automated drivi...

Towards Robust On-Ramp Merging via Augmented Multimodal Reinforcement Learning

Despite the success of AI-enabled onboard perception, on-ramp merging ha...

Quadratic Q-network for Learning Continuous Control for Autonomous Vehicles

Reinforcement Learning algorithms have recently been proposed to learn t...

A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers

Lane change is a crucial vehicle maneuver which needs coordination with ...

Maneuver Control based on Reinforcement Learning for Automated Vehicles in An Interactive Environment

Operating a robot safely and efficiently can be considerably challenging...

High-level Decisions from a Safe Maneuver Catalog with Reinforcement Learning for Safe and Cooperative Automated Merging

Reinforcement learning (RL) has recently been used for solving challengi...

An Interaction-aware Evaluation Method for Highly Automated Vehicles

It is important to build a rigorous verification and validation (V V) ...