Lazy Restless Bandits for Decision Making with Limited Observation Capability: Applications in Wireless Networks

01/04/2018
by   Kesav Kaza, et al.
0

In this work we formulate the problem of restless multi-armed bandits with cumulative feedback and partially observable states. We call these bandits as lazy restless bandits (LRB) as they are slow in action and allow multiple system state transitions during every decision interval. Rewards for each action are state dependent. The states of arms are hidden from the decision maker. The goal of the decision maker is to choose one of the M arms, at the beginning of each decision interval, such that long term cumulative reward is maximized. This work is motivated from applications in wireless networks such as relay selection, opportunistic channel access and downlink scheduling under evolving channel conditions. The Whittle index policy for solving LRB problem is analyzed. In course of doing so, various structural properties of the value functions are proved. Further, closed form index expressions are provided for two sets of special cases; for general cases, an algorithm for index computation is provided. A comparative study based on extensive numerical simulations is presented; the performances of Whittle index policy and myopic policy are compared with other policies such as uniform random, non-uniform random and round-robin.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/18/2019

Sequential Decision Making under Uncertainty with Dynamic Resource Constraints

This paper studies a class of constrained restless multi-armed bandits. ...
research
07/25/2020

Simulation Based Algorithms for Markov Decision Processes and Multi-Action Restless Bandits

We consider multi-dimensional Markov decision processes and formulate a ...
research
07/30/2021

Indexability and Rollout Policy for Multi-State Partially Observable Restless Bandits

Restless multi-armed bandits with partially observable states has applic...
research
07/20/2016

On the Identification and Mitigation of Weaknesses in the Knowledge Gradient Policy for Multi-Armed Bandits

The Knowledge Gradient (KG) policy was originally proposed for online ra...
research
05/30/2022

Optimistic Whittle Index Policy: Online Learning for Restless Bandits

Restless multi-armed bandits (RMABs) extend multi-armed bandits to allow...
research
07/21/2023

Bandits with Deterministically Evolving States

We propose a model for learning with bandit feedback while accounting fo...
research
10/31/2022

Indexability is Not Enough for Whittle: Improved, Near-Optimal Algorithms for Restless Bandits

We study the problem of planning restless multi-armed bandits (RMABs) wi...

Please sign up or login with your details

Forgot password? Click here to reset