DeepAI AI Chat
Log In Sign Up

Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning

by   Hengrui Cai, et al.

Evaluating the performance of an ongoing policy plays a vital role in many areas such as medicine and economics, to provide crucial instruction on the early-stop of the online experiment and timely feedback from the environment. Policy evaluation in online learning thus attracts increasing attention by inferring the mean outcome of the optimal policy (i.e., the value) in real-time. Yet, such a problem is particularly challenging due to the dependent data generated in the online environment, the unknown optimal policy, and the complex exploration and exploitation trade-off in the adaptive experiment. In this paper, we aim to overcome these difficulties in policy evaluation for online learning. We explicitly derive the probability of exploration that quantifies the probability of exploring the non-optimal actions under commonly used bandit algorithms. We use this probability to conduct valid inference on the online conditional mean estimator under each action and develop the doubly robust interval estimation (DREAM) method to infer the value under the estimated optimal policy in online learning. The proposed value estimator provides double protection on the consistency and is asymptotically normal with a Wald-type confidence interval provided. Extensive simulations and real data applications are conducted to demonstrate the empirical validity of the proposed DREAM method.


page 1

page 2

page 3

page 4


Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support

We address policy learning with logged data in contextual bandits. Curre...

Optimal Testing in the Experiment-rich Regime

Motivated by the widespread adoption of large-scale A/B testing in indus...

Jump Interval-Learning for Individualized Decision Making

An individualized decision rule (IDR) is a decision function that assign...

VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

Trading off exploration and exploitation in an unknown environment is ke...

Counterfactual Inference of the Mean Outcome under a Convergence of Average Logging Probability

Adaptive experiments, including efficient average treatment effect estim...

Policy Evaluation with Latent Confounders via Optimal Balance

Evaluating novel contextual bandit policies using logged data is crucial...

Reinforcement Learning under Model Risk for Biomanufacturing Fermentation Control

In the biopharmaceutical manufacturing, fermentation process plays a cri...