Human-in-the-Loop Synthesis for Partially Observable Markov Decision Processes

by   Steven Carr, et al.

We study planning problems where autonomous agents operate inside environments that are subject to uncertainties and not fully observable. Partially observable Markov decision processes (POMDPs) are a natural formal model to capture such problems. Because of the potentially huge or even infinite belief space in POMDPs, synthesis with safety guarantees is, in general, computationally intractable. We propose an approach that aims to circumvent this difficulty: in scenarios that can be partially or fully simulated in a virtual environment, we actively integrate a human user to control an agent. While the user repeatedly tries to safely guide the agent in the simulation, we collect data from the human input. Via behavior cloning, we translate the data into a strategy for the POMDP. The strategy resolves all nondeterminism and non-observability of the POMDP, resulting in a discrete-time Markov chain (MC). The efficient verification of this MC gives quantitative insights into the quality of the inferred human strategy by proving or disproving given system specifications. For the case that the quality of the strategy is not sufficient, we propose a refinement method using counterexamples presented to the human. Experiments show that by including humans into the POMDP verification loop we improve the state of the art by orders of magnitude in terms of scalability.


page 1

page 2

page 3

page 4


Motion Planning under Partial Observability using Game-Based Abstraction

We study motion planning problems where agents move inside environments ...

Counterexample-Guided Strategy Improvement for POMDPs Using Recurrent Neural Networks

We study strategy synthesis for partially observable Markov decision pro...

Barrier Functions for Multiagent-POMDPs with DTL Specifications

Multi-agent partially observable Markov decision processes (MPOMDPs) pro...

Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints

Recurrent neural networks (RNNs) have emerged as an effective representa...

Online Active Perception for Partially Observable Markov Decision Processes with Limited Budget

Active perception strategies enable an agent to selectively gather infor...

The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Models

Partially Observable Markov Decision Processes (POMDPs) are useful tools...

Evaluating Hierarchies through A Partially Observable Markov Decision Processes Methodology

Hierarchical clustering has been shown to be valuable in many scenarios,...

Please sign up or login with your details

Forgot password? Click here to reset