Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations

06/02/2023

∙

In real-world reinforcement learning (RL) systems, various forms of impaired observability can complicate matters. These situations arise when an agent is unable to observe the most recent state of the system due to latency or lossy channels, yet the agent must still make real-time decisions. This paper introduces a theoretical investigation into efficient RL in control systems where agents must act with delayed and missing state observations. We establish near-optimal regret bounds, of the form 𝒪̃(√( poly(H) SAK)), for RL in both the delayed and missing observation settings. Despite impaired observability posing significant challenges to the policy class and planning, our results demonstrate that learning remains efficient, with the regret bound optimally depending on the state-action size of the original system. Additionally, we provide a characterization of the performance of the optimal policy under impaired observability, comparing it to the optimal value obtained with full observability.

READ FULL TEXT

Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations

Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policies

Value Driven Representation for Human-in-the-Loop Reinforcement Learning

Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act

On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness

Real-time calibration of coherent-state receivers: learning by trial and error

Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations

Related Research

Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policies

Value Driven Representation for Human-in-the-Loop Reinforcement Learning

Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act

On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness

Real-time calibration of coherent-state receivers: learning by trial and error