Lazy Restless Bandits for Decision Making with Limited Observation Capability: Applications in Wireless Networks

by   Kesav Kaza, et al.

In this work we formulate the problem of restless multi-armed bandits with cumulative feedback and partially observable states. We call these bandits as lazy restless bandits (LRB) as they are slow in action and allow multiple system state transitions during every decision interval. Rewards for each action are state dependent. The states of arms are hidden from the decision maker. The goal of the decision maker is to choose one of the M arms, at the beginning of each decision interval, such that long term cumulative reward is maximized. This work is motivated from applications in wireless networks such as relay selection, opportunistic channel access and downlink scheduling under evolving channel conditions. The Whittle index policy for solving LRB problem is analyzed. In course of doing so, various structural properties of the value functions are proved. Further, closed form index expressions are provided for two sets of special cases; for general cases, an algorithm for index computation is provided. A comparative study based on extensive numerical simulations is presented; the performances of Whittle index policy and myopic policy are compared with other policies such as uniform random, non-uniform random and round-robin.



There are no comments yet.


page 1

page 2

page 3

page 4


Sequential Decision Making under Uncertainty with Dynamic Resource Constraints

This paper studies a class of constrained restless multi-armed bandits. ...

Simulation Based Algorithms for Markov Decision Processes and Multi-Action Restless Bandits

We consider multi-dimensional Markov decision processes and formulate a ...

Indexability and Rollout Policy for Multi-State Partially Observable Restless Bandits

Restless multi-armed bandits with partially observable states has applic...

On the Identification and Mitigation of Weaknesses in the Knowledge Gradient Policy for Multi-Armed Bandits

The Knowledge Gradient (KG) policy was originally proposed for online ra...

Thompson Sampling with Information Relaxation Penalties

We consider a finite time horizon multi-armed bandit (MAB) problem in a ...

Collapsing Bandits and Their Application to Public Health Interventions

We propose and study Collpasing Bandits, a new restless multi-armed band...

Scheduling in Wireless Networks with Spatial Reuse of Spectrum as Restless Bandits

We study the problem of scheduling packet transmissions with the aim of ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.