A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning

05/20/2021
by   YongFeng Li, et al.
0

In this paper, we consider the linear programming (LP) formulation for deep reinforcement learning. The number of the constraints depends on the size of state and action spaces, which makes the problem intractable in large or continuous environments. The general augmented Lagrangian method suffers the double-sampling obstacle in solving the LP. Namely, the conditional expectations originated from the constraint functions and the quadratic penalties in the augmented Lagrangian function impose difficulties in sampling and evaluation. Motivated from the updates of the multipliers, we overcome the obstacles in minimizing the augmented Lagrangian function by replacing the intractable conditional expectations with the multipliers. Therefore, a deep parameterized augment Lagrangian method is proposed. Furthermore, the replacement provides a promising breakthrough to integrate the two steps in the augmented Lagrangian method into a single constrained problem. A general theoretical analysis shows that the solutions generated from a sequence of the constrained optimizations converge to the optimal solution of the LP if the error is controlled properly. A theoretical analysis on the quadratic penalty algorithm under neural tangent kernel setting shows the residual can be arbitrarily small if the parameter in network and optimization algorithm is chosen suitably. Preliminary experiments illustrate that our method is competitive to other state-of-the-art algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/15/2019

Augmented Lagrangian Method for Thin Plates with Signorini Boundaries

We consider C^1-continuous approximations of the Kirchhoff plate problem...
research
04/09/2018

Frank-Wolfe Splitting via Augmented Lagrangian Method

Minimizing a function over an intersection of convex sets is an importan...
research
09/15/2020

Training neural networks under physical constraints using a stochastic augmented Lagrangian approach

We investigate the physics-constrained training of an encoder-decoder ne...
research
11/21/2017

First-order methods for constrained convex programming based on linearized augmented Lagrangian function

First-order methods have been popularly used for solving large-scale pro...
research
08/30/2013

Separable Approximations and Decomposition Methods for the Augmented Lagrangian

In this paper we study decomposition methods based on separable approxim...
research
01/15/2021

Constraint Handling in Continuous-Time DDP-Based Model Predictive Control

The Sequential Linear Quadratic (SLQ) algorithm is a continuous-time var...
research
02/18/2020

Local Propagation in Constraint-based Neural Network

In this paper we study a constraint-based representation of neural netwo...

Please sign up or login with your details

Forgot password? Click here to reset