Deconfounding Actor-Critic Network with Policy Adaptation for Dynamic Treatment Regimes

by   Changchang Yin, et al.
The Ohio State University

Despite intense efforts in basic and clinical research, an individualized ventilation strategy for critically ill patients remains a major challenge. Recently, dynamic treatment regime (DTR) with reinforcement learning (RL) on electronic health records (EHR) has attracted interest from both the healthcare industry and machine learning research community. However, most learned DTR policies might be biased due to the existence of confounders. Although some treatment actions non-survivors received may be helpful, if confounders cause the mortality, the training of RL models guided by long-term outcomes (e.g., 90-day mortality) would punish those treatment actions causing the learned DTR policies to be suboptimal. In this study, we develop a new deconfounding actor-critic network (DAC) to learn optimal DTR policies for patients. To alleviate confounding issues, we incorporate a patient resampling module and a confounding balance module into our actor-critic framework. To avoid punishing the effective treatment actions non-survivors received, we design a short-term reward to capture patients' immediate health state changes. Combining short-term with long-term rewards could further improve the model performance. Moreover, we introduce a policy adaptation method to successfully transfer the learned model to new-source small-scale datasets. The experimental results on one semi-synthetic and two different real-world datasets show the proposed model outperforms the state-of-the-art models. The proposed model provides individualized treatment decisions for mechanical ventilation that could improve patient outcomes.


page 1

page 2

page 3

page 4


The Actor-Advisor: Policy Gradient With Off-Policy Advice

Actor-critic algorithms learn an explicit policy (actor), and an accompa...

Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

Dynamic treatment recommendation systems based on large-scale electronic...

Actor-Critic Model Predictive Control

Despite its success, Model Predictive Control (MPC) often requires inten...

An Algorithmic Theory of Metacognition in Minds and Machines

Humans sometimes choose actions that they themselves can identify as sub...

A Geometric Perspective on Self-Supervised Policy Adaptation

One of the most challenging aspects of real-world reinforcement learning...

Offline reinforcement learning with uncertainty for treatment strategies in sepsis

Guideline-based treatment for sepsis and septic shock is difficult becau...

Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation

Reinforcement learning (RL) algorithms usually require a substantial amo...

Please sign up or login with your details

Forgot password? Click here to reset