Sufficient Exploration for Convex Q-learning

10/17/2022
by   Fan Lu, et al.
0

In recent years there has been a collective research effort to find new formulations of reinforcement learning that are simultaneously more efficient and more amenable to analysis. This paper concerns one approach that builds on the linear programming (LP) formulation of optimal control of Manne. A primal version is called logistic Q-learning, and a dual variant is convex Q-learning. This paper focuses on the latter, while building bridges with the former. The main contributions follow: (i) The dual of convex Q-learning is not precisely Manne's LP or a version of logistic Q-learning, but has similar structure that reveals the need for regularization to avoid over-fitting. (ii) A sufficient condition is obtained for a bounded solution to the Q-learning LP. (iii) Simulation studies reveal numerical challenges when addressing sampled-data systems based on a continuous time model. The challenge is addressed using state-dependent sampling. The theory is illustrated with applications to examples from OpenAI gym. It is shown that convex Q-learning is successful in cases where standard Q-learning diverges, such as the LQR problem.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/10/2023

Convex Q Learning in a Stochastic Environment: Extended Version

The paper introduces the first formulation of convex Q-learning for Mark...
research
11/04/2019

Persistency of Linear Programming Formulations for the Stable Set Problem

The Nemhauser-Trotter theorem states that the standard linear programmin...
research
09/10/2020

Analysis of Theoretical and Numerical Properties of Sequential Convex Programming for Continuous-Time Optimal Control

Through the years, Sequential Convex Programming (SCP) has gained great ...
research
10/14/2022

Model-Free Characterizations of the Hamilton-Jacobi-Bellman Equation and Convex Q-Learning in Continuous Time

Convex Q-learning is a recent approach to reinforcement learning, motiva...
research
12/13/2020

Uniform Capacitated Facility Location Problems with Penalties/Outliers

In this paper, we present a framework to design approximation algorithms...
research
08/08/2020

Convex Q-Learning, Part 1: Deterministic Optimal Control

It is well known that the extension of Watkins' algorithm to general fun...
research
11/13/2015

Solving Jigsaw Puzzles with Linear Programming

We propose a novel Linear Program (LP) based formula- tion for solving j...

Please sign up or login with your details

Forgot password? Click here to reset