Reinforcement Learning with Almost Sure Constraints

12/09/2021
by   Agustin Castellano, et al.
0

In this work we address the problem of finding feasible policies for Constrained Markov Decision Processes under probability one constraints. We argue that stationary policies are not sufficient for solving this problem, and that a rich class of policies can be found by endowing the controller with a scalar quantity, so called budget, that tracks how close the agent is to violating the constraint. We show that the minimal budget required to act safely can be obtained as the smallest fixed point of a Bellman-like operator, for which we analyze its convergence properties. We also show how to learn this quantity when the true kernel of the Markov decision process is not known, while providing sample-complexity bounds. The utility of knowing this minimal budget relies in that it can aid in the search of optimal or near-optimal policies by shrinking down the region of the state space the agent must navigate. Simulations illustrate the different nature of probability one constraints against the typically used constraints in expectation.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

01/23/2019

Reinforcement Learning of Markov Decision Processes with Peak Constraints

In this paper, we consider reinforcement learning of Markov Decision Pro...
11/22/2017

Budget Allocation in Binary Opinion Dynamics

In this article we study the allocation of a budget to promote an opinio...
02/27/2019

Learning Factored Markov Decision Processes with Unawareness

Methods for learning and planning in sequential decision problems often ...
12/24/2020

Assured RL: Reinforcement Learning with Almost Sure Constraints

We consider the problem of finding optimal policies for a Markov Decisio...
11/02/2020

Searching k-Optimal Goals for an Orienteering Problem on a Specialized Graph with Budget Constraints

We propose a novel non-randomized anytime orienteering algorithm for fin...
09/22/2019

Faster saddle-point optimization for solving large-scale Markov decision processes

We consider the problem of computing optimal policies in average-reward ...
07/16/2018

Shielded Decision-Making in MDPs

A prominent problem in artificial intelligence and machine learning is t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.