Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care

by   Ali Shirali, et al.

Most medical treatment decisions are sequential in nature. Hence, there is substantial hope that reinforcement learning may make it possible to formulate precise data-driven treatment plans. However, a key challenge for most applications in this field is the sparse nature of primarily mortality-based reward functions, leading to decreased stability of offline estimates. In this work, we introduce a deep Q-learning approach able to obtain more reliable critical care policies. This method integrates relevant but noisy intermediate biomarker signals into the reward specification, without compromising the optimization of the main outcome of interest (e.g. patient survival). We achieve this by first pruning the action set based on all available rewards, and second training a final model based on the sparse main reward but with a restricted action set. By disentangling accurate and approximated rewards through action pruning, potential distortions of the main objective are minimized, all while enabling the extraction of valuable information from intermediate signals that can guide the learning process. We evaluate our method in both off-policy and offline settings using simulated environments and real health records of patients in intensive care units. Our empirical results indicate that pruning significantly reduces the size of the action space while staying mostly consistent with the actions taken by physicians, outperforming the current state-of-the-art offline reinforcement learning method conservative Q-learning. Our work is a step towards developing reliable policies by effectively harnessing the wealth of available information in data-intensive critical care environments.


page 1

page 2

page 3

page 4


A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies

Sepsis is a leading cause of mortality and its treatment is very expensi...

Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space

Sepsis is a leading cause of death in the ICU. It is a disease requiring...

A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units

The management of invasive mechanical ventilation, and the regulation of...

Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning

Mechanical ventilation is a key form of life support for patients with p...

Deep Offline Reinforcement Learning for Real-World Treatment Optimization Applications

There is increasing interest in data-driven approaches for dynamically c...

An Optimal Policy for Patient Laboratory Tests in Intensive Care Units

Laboratory testing is an integral tool in the management of patient care...

Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care

Dental disease is one of the most common chronic diseases despite being ...

Please sign up or login with your details

Forgot password? Click here to reset