Assessment of Reward Functions for Reinforcement Learning Traffic Signal Control under Real-World Limitations

by   Alvaro Cabrejas Egea, et al.

Adaptive traffic signal control is one key avenue for mitigating the growing consequences of traffic congestion. Incumbent solutions such as SCOOT and SCATS require regular and time-consuming calibration, can't optimise well for multiple road use modalities, and require the manual curation of many implementation plans. A recent alternative to these approaches are deep reinforcement learning algorithms, in which an agent learns how to take the most appropriate action for a given state of the system. This is guided by neural networks approximating a reward function that provides feedback to the agent regarding the performance of the actions taken, making it sensitive to the specific reward function chosen. Several authors have surveyed the reward functions used in the literature, but attributing outcome differences to reward function choice across works is problematic as there are many uncontrolled differences, as well as different outcome metrics. This paper compares the performance of agents using different reward functions in a simulation of a junction in Greater Manchester, UK, across various demand profiles, subject to real world constraints: realistic sensor inputs, controllers, calibrated demand, intergreen times and stage sequencing. The reward metrics considered are based on the time spent stopped, lost time, change in lost time, average speed, queue length, junction throughput and variations of these magnitudes. The performance of these reward functions is compared in terms of total waiting time. We find that speed maximisation resulted in the lowest average waiting times across all demand levels, displaying significantly better performance than other rewards previously introduced in the literature.



There are no comments yet.


page 3


Assessment of Reward Functions in Reinforcement Learning for Multi-Modal Urban Traffic Control under Real-World limitations

Reinforcement Learning is proving a successful tool that can manage urba...

Deep Reinforcement Learning for Adaptive Traffic Signal Control

Many existing traffic signal controllers are either simple adaptive cont...

DQN Control Solution for KDD Cup 2021 City Brain Challenge

We took part in the city brain challenge competition and achieved the 8t...

A Comparison of Reward Functions in Q-Learning Applied to a Cart Position Problem

Growing advancements in reinforcement learning has led to advancements i...

Maximizing the Total Reward via Reward Tweaking

In reinforcement learning, the discount factor γ controls the agent's ef...

Qualitative Differences Between Evolutionary Strategies and Reinforcement Learning Methods for Control of Autonomous Agents

In this paper we analyze the qualitative differences between evolutionar...

Moral reinforcement learning using actual causation

Reinforcement learning systems will to a greater and greater extent make...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.