Online Learning for Non-Stationary A/B Tests

02/14/2018
by   Andres Munoz Medina, et al.
0

The rollout of new versions of a feature in modern applications is a manual multi-stage process, as the feature is released to ever larger groups of users, while its performance is carefully monitored. This kind of A/B testing is ubiquitous, but suboptimal, as the monitoring requires heavy human intervention, is not guaranteed to capture consistent, but short-term fluctuations in performance, and is inefficient, as better versions take a long time to reach the full population. In this work we formulate this question as that of expert learning, and give a new algorithm Follow-The-Best-Interval, FTBI, that works in dynamic, non-stationary environments. Our approach is practical, simple, and efficient, and has rigorous guarantees on its performance. Finally, we perform a thorough evaluation on synthetic and real world datasets and show that our approach outperforms current state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2019

Cascading Non-Stationary Bandits: Online Learning to Rank in the Non-Stationary Cascade Model

Non-stationarity appears in many online applications such as web search ...
research
12/01/2020

Non-Stationary Latent Bandits

Users of recommender systems often behave in a non-stationary fashion, d...
research
06/15/2020

Piecewise-Stationary Off-Policy Optimization

Off-policy learning is a framework for evaluating and optimizing policie...
research
01/29/2021

Learning User Preferences in Non-Stationary Environments

Recommendation systems often use online collaborative filtering (CF) alg...
research
04/25/2023

Real-time Safety Assessment of Dynamic Systems in Non-stationary Environments: A Review of Methods and Techniques

Real-time safety assessment (RTSA) of dynamic systems is a critical task...
research
12/28/2017

Online Ensemble Multi-kernel Learning Adaptive to Non-stationary and Adversarial Environments

Kernel-based methods exhibit well-documented performance in various nonl...
research
09/05/2022

Online Decision Making for Trading Wind Energy

This paper proposes and develops a new algorithm for trading wind energy...

Please sign up or login with your details

Forgot password? Click here to reset