On Effective Parallelization of Monte Carlo Tree Search

06/15/2020
by   Anji Liu, et al.
0

Despite its groundbreaking success in Go and computer games, Monte Carlo Tree Search (MCTS) is computationally expensive as it requires a substantial number of rollouts to construct the search tree, which calls for effective parallelization. However, how to design effective parallel MCTS algorithms has not been systematically studied and remains poorly understood. In this paper, we seek to lay its first theoretical foundations, by examining the potential performance loss caused by parallelization when achieving a desired speedup. In particular, we focus on studying the conditions under which the performance loss (measured in excess regret) vanishes over time. To this end, we propose a general parallel MCTS framework that can be specialized to major existing parallel MCTS algorithms. We derive two necessary conditions for the algorithms covered by the general framework to have vanishing excess regret (i.e. excess regret converges to zero as the total number of rollouts grows). We demonstrate the effectiveness of the necessary conditions by showing that, for depth-2 search trees, the recently developed WU-UCT algorithm satisfies both necessary conditions and has provable vanishing excess regret. Finally, we perform empirical studies to closely examine the necessary conditions under the general tree search setting (with arbitrary tree depth). It shows that the topological discrepancy between the search trees constructed by the parallel and the sequential MCTS algorithms is the main reason for the performance loss.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/28/2018

P-MCGS: Parallel Monte Carlo Acyclic Graph Search

Recently, there have been great interests in Monte Carlo Tree Search (MC...
research
03/30/2020

Parallelization of Monte Carlo Tree Search in Continuous Domains

Monte Carlo Tree Search (MCTS) has proven to be capable of solving chall...
research
05/11/2015

Adapting Improved Upper Confidence Bounds for Monte-Carlo Tree Search

The UCT algorithm, which combines the UCB algorithm and Monte-Carlo Tree...
research
04/02/2017

Structured Parallel Programming for Monte Carlo Tree Search

In this paper, we present a new algorithm for parallel Monte Carlo tree ...
research
07/07/2023

On the convergence of dynamic implementations of Hamiltonian Monte Carlo and No U-Turn Samplers

There is substantial empirical evidence about the success of dynamic imp...
research
06/09/2017

Monte-Carlo Tree Search by Best Arm Identification

Recent advances in bandit tools and techniques for sequential learning a...

Please sign up or login with your details

Forgot password? Click here to reset