Proximal Policy Optimization with Relative Pearson Divergence

10/07/2020
by   Taisuke Kobayashi, et al.
0

Deep reinforcement learning (DRL) is one of the promising approaches for introducing robots into complicated environments. The recent remarkable progress of DRL stands on regularization of policy. By constraining the update of policy, DRL allows the policy to improve stably and efficiently. Among them, a popular method, named proximal policy optimization (PPO), has been introduced. PPO clips density ratio of the latest and baseline policies with a threshold, while its minimization target is unclear. As another problem of PPO, the symmetric threshold is given numerically while the density ratio itself is in asymmetric domain, thereby causing unbalanced regularization of the policy. This paper therefore proposes a new variant of PPO by considering a regularization problem of relative Pearson (RPE) divergence, so-called PPO-RPE. This regularization yields the clear minimization target, which constrains the latest policy to the baseline one. Through its analysis, the intuitive threshold-based design consistent with the asymmetry of the threshold and the domain of density ratio can be derived. Four benchmark tasks were simulated to compare PPO-RPE and the conventional methods. As a result, PPO-RPE outperformed the conventional methods on all the tasks in terms of the task performance by the learned policy.

READ FULL TEXT
research
03/18/2022

Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio

Deep reinforcement learning (DRL) is one of the promising approaches for...
research
12/13/2022

PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration

Proximal Policy Optimization (PPO) is a highly popular policy-based deep...
research
08/23/2020

Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning

Deep reinforcement learning (DRL) is one of the promising approaches to ...
research
09/24/2021

Combing Policy Evaluation and Policy Improvement in a Unified f-Divergence Framework

The framework of deep reinforcement learning (DRL) provides a powerful a...
research
03/22/2023

P^3O: Transferring Visual Representations for Reinforcement Learning via Prompting

It is important for deep reinforcement learning (DRL) algorithms to tran...
research
01/24/2023

DeepADMR: A Deep Learning based Anomaly Detection for MANET Routing

We developed DeepADMR, a novel neural anomaly detector for the deep rein...
research
10/21/2019

Regularization Matters in Policy Optimization

Deep Reinforcement Learning (Deep RL) has been receiving increasingly mo...

Please sign up or login with your details

Forgot password? Click here to reset