Online learning with feedback graphs and switching costs

by   Anshuka Rangi, et al.

We study online learning when partial feedback information is provided following every action of the learning process, and the learner incurs switching costs for changing his actions. In this setting, the feedback information system can be represented by a graph, and previous work provided the expected regret of the learner in the case of a clique (Expert setup), or disconnected single loops (Multi-Armed Bandits). We provide a lower bound on the expected regret in the partial information (PI) setting, namely for general feedback graphs ---excluding the clique. We show that all algorithms that are optimal without switching costs are necessarily sub-optimal in the presence of switching costs, which motivates the need to design new algorithms in this setup. We propose two novel algorithms: Threshold Based EXP3 and EXP3.SC. For the two special cases of symmetric PI setting and Multi-Armed-Bandits, we show that the expected regret of both algorithms is order optimal in the duration of the learning process with a pre-constant dependent on the feedback system. Additionally, we show that Threshold Based EXP3 is order optimal in the switching cost, whereas EXP3.SC is not. Finally, empirical evaluations show that Threshold Based EXP3 outperforms previous algorithm EXP3 SET in the presence of switching costs, and Batch EXP3 in the special setting of Multi-Armed Bandits with switching costs, where both algorithms are order optimal.


page 1

page 2

page 3

page 4


Bandits with Feedback Graphs and Switching Costs

We study the adversarial multi-armed bandit problem where partial observ...

Online learning over a finite action set with limited switching

This paper studies the value of switching actions in the Prediction From...

SpecWatch: A Framework for Adversarial Spectrum Monitoring with Unknown Statistics

In cognitive radio networks (CRNs), dynamic spectrum access has been pro...

Anomaly Search Over Many Sequences With Switching Costs

This paper considers the quickest search problem to identify anomalies a...

Staged Multi-armed Bandits

In this paper, we introduce a new class of reinforcement learning method...

Markov Game with Switching Costs

We study a general Markov game with metric switching costs: in each roun...

Risk-Averse Multi-Armed Bandits with Unobserved Confounders: A Case Study in Emotion Regulation in Mobile Health

In this paper, we consider a risk-averse multi-armed bandit (MAB) proble...

Please sign up or login with your details

Forgot password? Click here to reset