Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learning

by   Conor F. Hayes, et al.

In many risk-aware and multi-objective reinforcement learning settings, the utility of the user is derived from a single execution of a policy. In these settings, making decisions based on the average future returns is not suitable. For example, in a medical setting a patient may only have one opportunity to treat their illness. Making decisions using just the expected future returns – known in reinforcement learning as the value – cannot account for the potential range of adverse or positive outcomes a decision may have. Therefore, we should use the distribution over expected future returns differently to represent the critical information that the agent requires at decision time by taking both the future and accrued returns into consideration. In this paper, we propose two novel Monte Carlo tree search algorithms. Firstly, we present a Monte Carlo tree search algorithm that can compute policies for nonlinear utility functions (NLU-MCTS) by optimising the utility of the different possible returns attainable from individual policy executions, resulting in good policies for both risk-aware and multi-objective settings. Secondly, we propose a distributional Monte Carlo tree search algorithm (DMCTS) which extends NLU-MCTS. DMCTS computes an approximate posterior distribution over the utility of the returns, and utilises Thompson sampling during planning to compute policies in risk-aware and multi-objective settings. Both algorithms outperform the state-of-the-art in multi-objective reinforcement learning for the expected utility of the returns.


Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search

In many risk-aware and multi-objective reinforcement learning settings, ...

Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Making

In many real-world scenarios, the utility of a user is derived from the ...

Feature Acquisition using Monte Carlo Tree Search

Feature acquisition algorithms address the problem of acquiring informat...

Backtesting Systemic Risk Forecasts using Multi-Objective Elicitability

Backtesting risk measure forecasts requires identifiability (for model c...

Multi-Objective Coordination Graphs for the Expected Scalarised Returns with Generative Flow Models

Many real-world problems contain multiple objectives and agents, where a...

Service Selection using Predictive Models and Monte-Carlo Tree Search

This article proposes a method for automated service selection to improv...

How to Combine Tree-Search Methods in Reinforcement Learning

Finite-horizon lookahead policies are abundantly used in Reinforcement L...

Please sign up or login with your details

Forgot password? Click here to reset