Delays in Reinforcement Learning

by   Pierre Liotet, et al.

Delays are inherent to most dynamical systems. Besides shifting the process in time, they can significantly affect their performance. For this reason, it is usually valuable to study the delay and account for it. Because they are dynamical systems, it is of no surprise that sequential decision-making problems such as Markov decision processes (MDP) can also be affected by delays. These processes are the foundational framework of reinforcement learning (RL), a paradigm whose goal is to create artificial agents capable of learning to maximise their utility by interacting with their environment. RL has achieved strong, sometimes astonishing, empirical results, but delays are seldom explicitly accounted for. The understanding of the impact of delay on the MDP is limited. In this dissertation, we propose to study the delay in the agent's observation of the state of the environment or in the execution of the agent's actions. We will repeatedly change our point of view on the problem to reveal some of its structure and peculiarities. A wide spectrum of delays will be considered, and potential solutions will be presented. This dissertation also aims to draw links between celebrated frameworks of the RL literature and the one of delays.


page 1

page 2

page 3

page 4


Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays

Several real-world scenarios, such as remote control and sensing, are co...

Resolving Congestions in the Air Traffic Management Domain via Multiagent Reinforcement Learning Methods

In this article, we report on the efficiency and effectiveness of multia...

Reactive Reinforcement Learning in Asynchronous Environments

The relationship between a reinforcement learning (RL) agent and an asyn...

On the Analysis of Computational Delays in Reinforcement Learning-based Rate Adaptation Algorithms

Several research works have applied Reinforcement Learning (RL) algorith...

Enhancing reinforcement learning by a finite reward response filter with a case study in intelligent structural control

In many reinforcement learning (RL) problems, it takes some time until a...

Machine Learning assisted Chimera and Solitary states in Networks

Chimera and Solitary states have captivated scientists and engineers due...

Neural Laplace Control for Continuous-time Delayed Systems

Many real-world offline reinforcement learning (RL) problems involve con...

Please sign up or login with your details

Forgot password? Click here to reset