Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs

02/14/2023
by   Stefan Werner, et al.
0

The goal of this paper is to make a strong point for the usage of dynamical models when using reinforcement learning (RL) for feedback control of dynamical systems governed by partial differential equations (PDEs). To breach the gap between the immense promises we see in RL and the applicability in complex engineering systems, the main challenges are the massive requirements in terms of the training data, as well as the lack of performance guarantees. We present a solution for the first issue using a data-driven surrogate model in the form of a convolutional LSTM with actuation. We demonstrate that learning an actuated model in parallel to training the RL agent significantly reduces the total amount of required data sampled from the real system. Furthermore, we show that iteratively updating the model is of major importance to avoid biases in the RL training. Detailed ablation studies reveal the most important ingredients of the modeling process. We use the chaotic Kuramoto-Sivashinsky equation do demonstarte our findings.

READ FULL TEXT

page 1

page 6

page 7

page 11

research
01/30/2017

Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning

This article shows how the recent breakthroughs in Reinforcement Learnin...
research
06/13/2018

Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control

Recent work has shown that reinforcement learning (RL) is a promising ap...
research
01/25/2023

Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learning

We present a convolutional framework which significantly reduces the com...
research
05/13/2022

Deep Reinforcement Learning for Computational Fluid Dynamics on HPC Systems

Reinforcement learning (RL) is highly suitable for devising control stra...
research
05/01/2022

Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning

Deep reinforcement learning (RL) is a data-driven method capable of disc...
research
11/21/2020

On the Convergence of Reinforcement Learning

We consider the problem of Reinforcement Learning for nonlinear stochast...
research
03/29/2022

Deep Reinforcement Learning for Data-Driven Adaptive Scanning in Ptychography

We present a method that lowers the dose required for a ptychographic re...

Please sign up or login with your details

Forgot password? Click here to reset