DeepAI AI Chat
Log In Sign Up

Exploring Optimal Control With Observations at a Cost

by   Rui Aguiar, et al.

There has been a current trend in reinforcement learning for healthcare literature, where in order to prepare clinical datasets, researchers will carry forward the last results of the non-administered test known as the last-observation-carried-forward (LOCF) value to fill in gaps, assuming that it is still an accurate indicator of the patient's current state. These values are carried forward without maintaining information about exactly how these values were imputed, leading to ambiguity. Our approach models this problem using OpenAI Gym's Mountain Car and aims to address when to observe the patient's physiological state and partly how to intervene, as we have assumed we can only act after following an observation. So far, we have found that for a last-observation-carried-forward implementation of the state space, augmenting the state with counters for each state variable tracking the time since last observation was made, improves the predictive performance of an agent, supporting the notion of "informative missingness", and using a neural network based Dynamics Model to predict the most probable next state value of non-observed state variables instead of carrying forward the last observed value through LOCF further improves the agent's performance, leading to faster convergence and reduced variance.


The Concept of Criticality in Reinforcement Learning

Reinforcement learning methods carry a well known bias-variance trade-of...

Learning to Predict Without Looking Ahead: World Models Without Forward Prediction

Much of model-based reinforcement learning involves learning a model of ...

Visualizing MuZero Models

MuZero, a model-based reinforcement learning algorithm that uses a value...

SIDE: I Infer the State I Want to Learn

As one of the solutions to the Dec-POMDP problem, the value decompositio...

Learning to act: a Reinforcement Learning approach to recommend the best next activities

The rise of process data availability has led in the last decade to the ...

Embedding Positive Process Models into Lognormal Bayesian State Space Frameworks using Moment Matching

In ecology it is common for processes to be bounded based on physical co...