LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning

09/21/2022
by   Hosein Hasanbeig, et al.
0

LCRL is a software tool that implements model-free Reinforcement Learning (RL) algorithms over unknown Markov Decision Processes (MDPs), synthesising policies that satisfy a given linear temporal specification with maximal probability. LCRL leverages partially deterministic finite-state machines known as Limit Deterministic Buchi Automata (LDBA) to express a given linear temporal specification. A reward function for the RL algorithm is shaped on-the-fly, based on the structure of the LDBA. Theoretical guarantees under proper assumptions ensure the convergence of the RL algorithm to an optimal policy that maximises the satisfaction probability. We present case studies to demonstrate the applicability, ease of use, scalability, and performance of LCRL. Owing to the LDBA-guided exploration and LCRL model-free architecture, we observe robust performance, which also scales well when compared to standard RL approaches (whenever applicable to LTL specifications). Full instructions on how to execute all the case studies in this paper are provided on a GitHub page that accompanies the LCRL distribution www.github.com/grockious/lcrl.

READ FULL TEXT

page 3

page 8

research
05/02/2023

Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees

Linear Temporal Logic (LTL) is widely used to specify high-level objecti...
research
05/04/2020

Formal Policy Synthesis for Continuous-Space Systems via Reinforcement Learning

This paper studies data-driven techniques for satisfying temporal proper...
research
02/02/2019

Certified Reinforcement Learning with Logic Guidance

This paper proposes the first model-free Reinforcement Learning (RL) fra...
research
03/26/2021

Model-Free Learning of Safe yet Effective Controllers

In this paper, we study the problem of learning safe control policies th...
research
02/08/2021

Learning Optimal Strategies for Temporal Tasks in Stochastic Games

Linear temporal logic (LTL) is widely used to formally specify complex t...
research
10/02/2020

Model-Free Reinforcement Learning for Stochastic Games with Linear Temporal Logic Objectives

We study the problem of synthesizing control strategies for Linear Tempo...
research
09/09/2023

Verifiable Reinforcement Learning Systems via Compositionality

We propose a framework for verifiable and compositional reinforcement le...

Please sign up or login with your details

Forgot password? Click here to reset