On Minimax Optimal Offline Policy Evaluation

1 Introduction

In reinforcement learning, one of the most fundamental problems is

policy evaluation — estimate the average reward obtained by running a given policy to select actions in an unknown system. A straightforward solution is to simply run the policy and measure the rewards it collects. In many applications, however, running a new policy in the actual system can be expensive or even impossible. For example, flying a helicopter with a new policy can be risky as it may lead to crashes; deploying a new ad display policy on a website may be catastrophic to user experience; testing a new treatment on patients may simply be impossible for legal and ethical reasons; etc.

These difficulties make it critical to do off-policy policy evaluation (Precup et al., 2000, Sutton et al., 2010), which is sometimes referred to as offline evaluation in the bandit literature (Li et al., 2011) or counterfactual reasoning (Bottou et al., 2013). Here, we still aim to estimate the average reward of a target policy, but instead of being able to run the policy online, we only have access to a sample of observations made about the unknown system, which may be collected in the past using a different policy. Off-policy evaluation has been found useful in a number of important applications (Langford et al., 2008, Li et al., 2011, Bottou et al., 2013) and can also be looked as a key building block for policy optimization which, as in supervised learning, can often be reduced to evaluation, as long as the complexity of the policy class is well-controlled (Ng and Jordan, 2000). For example, it has played an important role in many optimization algorithms for Markov decision processes (e.g., Heidrich-Meisner and Igel 2009) and bandit problems (Auer et al., 2002, Langford and Zhang, 2008, Strehl et al., 2011). ^{size=,color=red!20!white,}^{size=,color=red!20!white,}todo: size=,color=red!20!white,Lihong: Others? Examples for partial-monitor games? In the context of supervised learning, in the covariate shift literature, the problem of estimating losses under changing distributions is crucial for model selection (Sugiyama and Müller, 2005, Yu and Szepesvári, 2012)

and also appears in active learning

(Dasgupta, 2011). In the statistical literature, on the other hand, the problem appears in the context of randomized experiments. Here, the focus is on the two-action (binary) case where the goal is to estimate the difference between the expected rewards of the two actions (Hirano et al., 2003), which is slightly (but not essentially) different than our setting. ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: Mention in conclusion generalization

The topic of the present paper is off-policy evaluation in finite settings, under a mean squared error criterion (MSE). As opposed to the statistics literature (Hirano et al., 2003), we are interested in results for finite sample sizes. In particular, we are interested in limits of performance (minimax MSE) given fixed policies, but unknown stochastic rewards with bounded mean reward, as well as the performance of estimation procedures compared to the minimax MSE. We argue that the finite setting is not a key limitation when focusing on the scaling behavior of the MSE of algorithms. Moreover, we are not aware of prior work that would have studied the above problem (i.e., relating the MSE of algorithms to the best possible MSE). Our main results are as follows: We start with a lower bound on the minimax MSE, to set a target for the estimation procedures. Next, we derive the exact MSE of the likelihood ratio (or importance-weighted) estimator (LR), which is shown to have an extra (uncontrollable) factor as compared to the minimax MSE lower bound. Next, we consider the estimator which estimates the mean rewards by sample means, which we call the regression estimator (REG). The motivation of studying this estimator is both its simplicity and also because it is known that a related estimator is asymptotically efficient (Hirano et al., 2003). The main question is whether the asymptotic efficiency transfers into finite-time efficiency. Our answer to this is mixed: We show that the MSE of REG is within a constant factor of the minimax MSE lower bound, however, the “constant” depends on the number of actions ( $K$

), or a lower bound on the variance. We also show that the dependence of the MSE of REG on the number actions is unavoidable. In any case, for “small” action sets or high noise setting, the REG estimator can be thought of as a minimax near-optimal estimator. We also show that for small sample sizes (up to

\sqrt{K}

) all estimators must suffer a constant MSE. Numerical experiments illustrate the tightness of the analysis. Implications for more complicated settings, such as policy evaluation in contextual bandits and Markov Decision Processes (MDPs). The question of designing a nearly minimax estimator independently of any problem parameters remains open. All the proofs ot given in the main text can be found in the supplementary material.

2 Multi-armed Bandit

Let $A = {1, 2, \dots, K}$ be a finite set of $K$ actions. Data $D^{n} = {(A_{i}, R_{i})}_{1 \leq i \leq n}$ is generated by the following process: ¹¹1The data $D^{n}$ is actually a list, not a set. We keep the notation ${(A_{i}, R_{i})}_{1 \leq i \leq n}$ for historical reasons. $(A_{i}, R_{i})$ are independent copies of $(A, R)$ , where $P (A = a) = π_{D} (a)$ and $R \sim Φ (\cdot | A)$ for some unknown family of distributions ${Φ (\cdot | a)}_{a \in A}$ and known policy $π_{D}$ . We are also given a known target policy $π$ and want to estimate its value, $v_{Φ}^{π} := E_{A \sim π, R \sim Φ (\cdot | A)} [R]$ based on the knowledge of $D^{n}$ , $π_{D}$ and $π$ , where the quality of an estimate $ˆ v$ constructed based on $D^{n}$ (and $π, π_{D}$ ) is measured by its mean-squared error, $M S E (ˆ v) := E [(ˆ v - v_{Φ}^{π})^{2}]$ .

Define $r_{Φ} (a) := E [R | A = a]$ and $σ_{Φ}^{2} (a) := V (R | A = a)$ , where $V (\cdot)$ stands for the variance. Further, let $π_{D}^{*} := {min}_{a} π_{D} (a)$ . For convenience, we will identify any function $f : A \to R$ with the $K$

-dimensional vector whose

k

th component is

f (k)

. Thus,

r_{Φ}

σ_{Φ}^{2}

, etc. will also be looked at as vectors. Note that we do not assume that the rewards are bounded from either direction. ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: Really? ^{size=,color=red!20!white,}^{size=,color=red!20!white,}todo: size=,color=red!20!white,Lihong: Where to mention

R_{m a x}

? Shall we assume

r \geq 0

here?

A few quantities are introduced to facilitate discussions that follow:

	$V_{1}$	$:=$	$E [V (\frac{π (A)}{π_{D} (A)} R \| A)] = \sum a \frac{π^{2} (a)}{π_{D} (a)} σ_{Φ}^{2} (a),$
	$V_{2}$	$:=$	$V (E [\frac{π (A)}{π_{D} (A)} R \| A]) = V (\frac{π (A)}{π_{D} (A)} r_{Φ} (A)) = \sum a \frac{π^{2} (a)}{π_{D} (a)} r_{Φ} (a)^{2} - (v_{Φ}^{π})^{2} .$

Note that $V_{1}$ and $V_{2}$ are functions of $Φ, π_{D}$ and $π$ , but this dependence is suppressed. Also, $V_{1}$ and $V_{2}$ are independent in that there are no constants $c, C > 0$ such that $c V_{1} \leq V_{2} \leq C V_{1}$ for any $π, π_{D}, Φ$ . Finally, let $p_{a, n} := (1 - π_{D} (a))^{n}$

be the probability of having

no sample of

a

D^{n}

2.1 A Minimax Lower Bound

We start with establishing a minimax lower bound that characterizes the inherent hardness of the off-policy evaluation problem. An estimator $A$ can be considered as a function that maps $(π, π_{D}, D^{n})$ to an estimate of $v_{Φ}^{π}$ , denoted ${ˆ v}_{A} (π, π_{D}, D^{n})$ . Fix $σ^{2} := (σ^{2} (a))_{a \in A}$ . We consider the minimax optimal risk subject to $σ_{Φ}^{2} (a) \leq σ^{2} (a)$ ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: Sometimes we use $V (Φ)$ , sometimes $σ_{Φ}^{2}$ . Unify the notation or tell the reader to anticipate this. and $0 \leq r_{Φ} (a) \leq R_{m a x}$ for all $a \in A$ :

R_{n}^{*} (π, π_{D}, R_{m a x}, σ^{2}) := inf A sup Φ : σ_{Φ}^{2} \leq σ^{2}, 0 \leq r_{Φ} \leq R_{m a x} E [({ˆ v}_{A} (π, π_{D}, D^{n}) - v_{Φ}^{π})^{2}],

where for vectors $x, y \in R^{K}$ , $x \leq y$ holds if and only if $x_{i} \leq y_{i}$ for $1 \leq i \leq K$ . For $B \subset A$ , we let $p_{B, n}$ denote the probability that none of the actions in the data $D^{n}$ falls into $B$ : $p_{B, n} = P (A_{1}, \dots, A_{n} \notin B)$ . Note that this definition generalizes $p_{a, n}$ . We also let $π (B) = \sum_{a \in B} π (a)$ .

Theorem 1.

For any $n > 0$ , $π_{D}$ , $π$ , $R_{m a x}$ and $σ^{2}$ , one has

R_{n}^{*} (π, π_{D}, R_{m a x}, σ^{2}) \geq \frac{1}{4} max (R_{m a x}^{2} max B \subset A π^{2} (B) p_{B, n}, \frac{V_{1}}{n}) .

Furthermore,

liminf n \to \infty \frac{R_{n}^{*} (π, π_{D}, R_{m a x}, σ^{2})}{V_{1} / n} \geq 1.

(1)

Proof.

To prove the first part of the lower bound, fix a subset $B \subset A$ of actions and choose an environment $Φ \in E$ , where $E$ is the set of environments $Φ$ such that $σ_{Φ}^{2} \leq σ^{2}$ and $0 \leq r_{Φ} \leq R_{m a x}$ . Introduce the notation $E_{Φ}$ to denote expectation when the data is generated by environment $Φ$ . ^{size=,color=red!20!white,}^{size=,color=red!20!white,}todo: size=,color=red!20!white,Lihong: Why call “environment”, not “distribution”?

Let $D^{n}$ be the data generated based on $π_{D}$ and $Φ$ and let ${ˆ v}_{A} (D^{n})$ denote the estimate produced by some algorithm $A$ . Define $S = {A_{1}, \dots, A_{n}}$ to be the set of actions in the dataset that is seen by the algorithm. Clearly, for any $Φ, Φ^{'}$ such that they agree on the complement of $B$ (but may differ on actions in $B$ ),^{size=,color=red!20!white,}^{size=,color=red!20!white,}todo: size=,color=red!20!white,Lihong: Agreement between $Φ$ and $Φ^{'}$ , not between $r_{Φ}$ and $r_{Φ^{'}}$ ?

E_{Φ} [{ˆ v}_{A} (D_{n}) | S \cap B = \emptyset] = E_{Φ^{'}} [{ˆ v}_{A} (D_{n}) | S \cap B = \emptyset] .

(2)

Now, ${M S E}_{Φ} ({ˆ v}_{A}) := E_{Φ} [({ˆ v}_{A} (D^{n}) - v_{Φ}^{π})^{2}] \geq E_{Φ} [({ˆ v}_{A} (D^{n}) - v_{Φ}^{π})^{2} | S \cap B = \emptyset] P (S \cap B = \emptyset)$ and by adapting the argument that the MSE is lower bounded by the bias squared, $E_{Φ} [({ˆ v}_{A} (D^{n}) - v_{Φ}^{π})^{2} | S \cap B = \emptyset] \geq (E_{Φ} [{ˆ v}_{A} (D^{n}) | S \cap B = \emptyset] - v_{Φ}^{π})^{2}$ . Hence, ${M S E}_{Φ} ({ˆ v}_{A}) \geq P (S \cap B = \emptyset) {sup}_{Φ \in E} (E_{Φ} [{ˆ v}_{A} (D^{n}) | S \cap B = \emptyset] - v_{Φ}^{π})^{2}$ . We get an even smaller quantity if we further restrict the environments $Φ$ to environments $E_{0}$ that also satisfy $r_{Φ} = σ_{Φ}^{2} = 0$ on $A ∖ B$ . Now, by (2), for all these environments, $E_{Φ} [{ˆ v}_{A} (D^{n}) | S \cap B = \emptyset]$ takes on a common value, denote it by $v_{A}$ . Hence, ${M S E}_{Φ} ({ˆ v}_{A}) \geq P (S \cap B = \emptyset) {sup}_{Φ \in E_{0}} (v_{A} - v_{Φ}^{π})^{2}$ . Since $v_{Φ}^{π} = \sum_{a \in B} π (a) r_{Φ} (a)$ , ${sup}_{Φ \in E_{0}} (v_{A} - v_{Φ}^{π})^{2} \geq \frac{R_{m a x}^{2}}{4} π^{2} (B)$ , where we use the shorthand $π (B) = \sum_{a \in B} π (a)$ . Plugging this into the previous inequality we get ${sup}_{Φ \in E} {M S E}_{Φ} ({ˆ v}_{A}) \geq P (S \cap B = \emptyset) \frac{R_{m a x}^{2}}{} 4 π^{2} (B)$ . Since $A$ was arbitrary, we get $R_{n}^{*} (π, π_{D}, R_{m a x}, σ^{2}) \geq P (S \cap B = \emptyset) \frac{R_{m a x}^{2}}{4} π^{2} (B)$ .

For the second part, consider a class of normal distributions with fixed reward variances

σ^{2}

but different reward expectations:

F_{p} = {Φ_{0}, \dots, Φ_{p - 1}}

, where

r_{Φ_{i}} = 2 i \sqrt{ε} Δ \in R^{K}

, for some to-be-specified vector

Δ \in R_{+}^{K}

that satisfies

\sum_{a} π (a) Δ (a) = 1

. The data-generating distribution

Φ

is in

F_{p}

, but is unknown otherwise.

It is easy to see that the policy value between any two distributions in $F_{p}$ differ by at least $2 \sqrt{ε}$ . Indeed, for any $Φ_{i}, Φ_{j} \in F_{p}$ , $| v_{Φ_{i}}^{π} - v_{Φ_{j}}^{π} | = 2 \sqrt{ε} | i - j | \sum_{a} π (a) Δ (a) = 2 \sqrt{ε} | i - j | \geq 2 \sqrt{ε}$ . It follows that, in order to achieve a squared error less than $ε$ , one needs to identify the underlying data-generating $Φ$ from $F_{p}$ , based on the observed sample $D^{n}$ . The problem now reduces to finding a minimax lower bound for hypothesis testing in the given finite set $F_{p}$ .

We resort to the information-theoretic machinery based on Fano’s inequality (see, e.g., Raginsky and Rakhlin (2011)). Define an oracle which, when queried, outputs $Y = (A, R)$ with $A \sim π_{D} (\cdot)$ and $R \sim Φ (\cdot | A)$ . Let the distribution of $Y$ when $Φ$ is used be denoted by $P_{Y | Φ}$ . Let $F_{p}$ collect $p$ distributions such that $Φ (\cdot | a)$ is normal. Consider $Φ, Φ^{'} \in F_{p}$ . Then,

D (P_{Y | Φ} ∥ P_{Y | Φ^{'}}) = \sum a π_{D} (a) D (Φ (\cdot | a) ∥ Φ^{'} (\cdot | a)) = 2 ε (i - j)^{2} \sum a \frac{π_{D} (a) Δ (a)^{2}}{σ (a)^{2}} .

The divergence measures how much information is carried in one sample from the oracle to tell $Φ$ from $Φ^{'}$ . To obtain the tightest lower bound, we should minimize the divergence. Subject to the constraint $\sum_{a} π (a) Δ (a) = 1$ , the divergence is minimized by setting $Δ (a) \propto \frac{π (a)}{π_{D} (a)} σ^{2} (a)$ , and is $2 ε (i - j)^{2} / V_{1}$ . Now setting $p = 6$ , and applying Lemma 1, Theorem 1 and the “Information Radius bound” from Raginsky and Rakhlin (2011), we have $n \geq \frac{V_{1}}{4 ε}$ . ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: Can we add these to the appendix? Reorganizing terms and combining with the first term complete the proof of the first statement.

For the second part, note that it suffices to consider asymptotically unbiased estimators (cf. the generalized Cramer-Rao lower bound, Theorem 7.3 of

Ibragimov and Has’minskii 1981). For any such estimator, the Cramer-Rao lower bound gives the result with the parametric family chosen to be

p (a, y; θ) = π_{D} (a) φ (y; r (a), σ^{2} (a))

, where

θ = (r (a))_{a \in A}

is the unknown parameter to be estimated, and

φ (\cdot; μ, σ^{2})

is the density of the normal distribution with mean

μ

and variance

σ^{2}

and the quantity to be estimated is

ψ (θ) = \sum_{a} π (a) r (a)

. For details, see Section A.1. ∎

The next corollary says that the minimax risk is constant when the number of samples is $O (\sqrt{K})$ :

Corollary 1.

For $K \geq 2$ , $n \leq \sqrt{K}$ , .

Proof.

Choose $B \subset A$ to minimize $π_{D} (B)$ subject to the constraint $| B | = ⌊ \sqrt{K} ⌋$ . Note that $P (A_{1}, \dots, A_{n} \notin B) = (1 - π_{D} (B))^{n} \geq (1 - | B | / K)^{n} \geq (1 - 1 / \sqrt{K})^{\sqrt{K}} \geq (1 - 1 / \sqrt{2})^{\sqrt{2}}$ . Choosing $π$ such that $π (B) = 1$ gives the result. ∎

We conjecture that the result can be strengthened by increasing the upper limit on $n$ . ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: One of the conjectures; mention in conclusion

2.2 Likelihood Ratio Estimator

One of the most popular estimators is known as the propensity score estimator in the statistical literature (Rosenbaum and Rubin, 1983, 1985), or the importance weighting estimator (Bottou et al., 2013). We call it the likelihood ratio estimator, as it estimates the unknown value using likelihood ratios, or importance weights:

{ˆ v}_{L R} (π, π_{D}, D^{n}) := \frac{1}{n} n \sum i = 1 \frac{π (A_{i})}{π_{D} (A_{i})} R_{i} .

Its distinguishing feature is that it is unbiased: $E [{ˆ v}_{L R} (π, π_{D}, D^{n})] = v_{Φ}^{π}$ , implying that the MSE is purely contributed by the variance of the estimator. The main result in this subsection shows that this estimator does not achieve the minimax lower bound up to any constant (by making $V_{2} ≫ V_{1}$ ). The proof (given in the appendix) is based on a direct calculation using the law of total variance.

Proposition 1.

It holds that $M S E ({ˆ v}_{L R} (π, π_{D}, D^{n})) = (V_{1} + V_{2}) / n$ .

We see that as compared to the lower bound on the minimax MSE, an extra $V_{2} / n$ factor appears. In the next section, we will see that this factor is superfluous, showing that the MSE of LR can be “unreasonably large”.

2.3 Regression Estimator

For convenience, define $n (a) := \sum_{i = 1}^{n} I (A_{i} = a)$ to be the number of samples for action $a$ in $D^{n}$ , and $R (a) := \sum_{i = 1}^{n} I (A_{i} = a) R_{i}$ the total rewards of $a$ . The regression estimator (REG) is given by

{ˆ v}_{R e g} (π, D^{n}) := \sum a π (a) ˆ r (a), % w h e r e ˆ r (a) := {\begin{matrix} 0, & if n (a) = 0; \frac{R (a)}{n (a)}, & otherwise . \end{matrix}

For brevity, we will also write $ˆ r (a) = I {n (a) > 0} \frac{R (a)}{n (a)}$ , where we take $\frac{0}{0}$ to be zero. The name of the estimator comes from the fact that it estimates the reward function, and the problem of estimating the reward function can be thought of as a regression problem.

Interestingly, as can be verified by direct calculation, the REG estimator can also be written as

{ˆ v}_{R e g} (π, D^{n}) = \frac{1}{n} n \sum i = 1 \frac{π (A_{i})}{{ˆ π}_{D} (A_{i})} R_{i},

(3)

where ${ˆ π}_{D} (a) = \frac{n (a)}{n}$ is the empirical estimate of $π_{D} (a)$ . Hence, the main difference between LR and REG is that the former uses $π_{D}$ to reweight the data, while the latter uses the empirical estimates ${ˆ π}_{D}$ . It may appear that LR is superior since it uses the “right” quantity. Surprisingly, REG turns out to be much more robust than LR, as will be shown shortly; further discussion is made in Section D.

For the next statement, the counterpart of creftype 1, the following quantities will be useful:

	$V_{0, n}$	$:= {(\sum a π (a) r_{Φ} (a) p_{a, n})}_{Φ}^{2} + \sum a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} (1 - p_{a, n}) and$
	$V_{3, n}$	$:= \sum a E [\frac{I {n (a) > 0}}{{ˆ π}_{D} (a)} - \frac{1}{π_{D} (a)}] π (a)^{2} σ^{2} (a) .$

Proposition 2.

Fix $π, π_{D}$ . Assume that $r_{Φ}$ is nonnegative valued. ^{size=,color=red!20!white,}^{size=,color=red!20!white,}todo: size=,color=red!20!white,Lihong: How about making it an assumption in the general setting? Then it holds that $M S E ({ˆ v}_{R e g} (π, D^{n})) \leq V_{0, n} + (V_{1} + V_{3, n}) / n$ . Further, for any $Φ$ such that the rewards have normal distributions, defining $b_{n} = \sum_{a} π (a) r_{Φ} (a) p_{a, n}$ to be the bias of ${ˆ v}_{R e g}$ , $M S E ({ˆ v}_{R e g}) \geq \frac{V_{1}}{n} + 4 b_{n}^{2} (1 + \frac{V_{1}}{n}) + \frac{2}{n} \sum_{a} \frac{π^{2} (a)}{π_{D} (a)} σ_{Φ}^{2} (a) p_{a, n}$ .

Proof sketch.

For the upper bound use that the MSE equals the sum of squared bias and the variance. It can be verified that REG is slightly biased: $E [{ˆ v}_{R e g}] - v_{Φ}^{π} = \sum_{a} π (a) r_{Φ} (a) p_{a, n}$ . For the variance term, we use the law of total variance to yield: $V ({ˆ v}_{R e g}) = E [V ({ˆ v}_{R e g} | n (1), \dots, n (K))] + V (E [{ˆ v}_{R e g} | n (1), \dots, n (K)])$ , where the first term is $\sum_{a} π^{2} (a) σ^{2} (a) E [I {n (a) > 0} / n (a)]$ , and the second term is upper bounded (Lemma 2) by $\sum_{a} π^{2} (a) r_{Φ}^{2} (a) p_{a, n} (1 - p_{a, n})$ . The proof is then completed by adding squared bias to variance, and using definitions of $V_{0, n}$ , $V_{1}$ , and $V_{3}$ . The lower bound follows from the (generalized) Cramer-Rao inequality. ∎

The main result of this section is the following theorem that characterizes the MSE of REG in terms of the minimax optimal MSE.

Theorem 2 (Minimax Optimality of the Regression Estimator).

The following hold:

For any $π, π_{D}$ , $σ^{2} = (σ^{2} (a))_{a \in A}$ , $Φ$ such that ${min}_{a} r_{Φ} (a) \geq 0$ , ${max}_{a} r_{Φ} (a) \leq R_{m a x}$ , and $σ_{Φ}^{2} \leq σ^{2}$ , it holds for any $n > 0$ that

$M S E ({ˆ v}_{R e g} (π, D_{n})) \leq K {min (4 K, max a \frac{r_{Φ}^{2} (a)}{σ_{Φ}^{2} (a)}) + 5} R_{n}^{*} (π, π_{D}, R_{m a x}, σ^{2}),$ (4)

where $D_{n} = {(A_{i}, R_{i})}_{i = 1, \dots, n}$ is an i.i.d. sample from $(π_{D}, Φ)$ .
A suboptimality factor of $Ω (K)$ in the above result is unavoidable: For $K > 2$ , there exists $(π, π_{D})$ such that for any $n \geq 1$ ,

$\frac{M S E ({ˆ v}_{R e g} (π, D_{n}))}{R_{n}^{*} (π, π_{D}, R_{m a x}, 0)} \geq n e^{- 2 n / (K - 1)} .$

Thus for $n = (K - 1) / 2$ , this ratio is at least $\frac{K - 1}{2 e}$ .
The estimator ${ˆ v}_{R e g}$ is asymptotically minimax optimal:

$limsup n \to \infty \frac{M S E ({ˆ v}_{R e g} (π, D_{n}))}{R_{n}^{*} (π, π_{D}, R_{m a x}, σ^{2})} \leq 1 .$

We need the following lemma, which may be of interest on its own:

Lemma 1.

Let $X_{1}, \dots, X_{n}$ be $n$

independent Bernoulli random variables with parameter

p > 0

. Letting

S_{n} = \sum_{i = 1}^{n} X_{i}

ˆ p = S_{n} / n

Z = \frac{I {S_{n} > 0}}{ˆ p} - \frac{1}{p}

, we have for any

n

and

p

that

E [Z] \leq 4 / p

. Further, when

n p \geq 34

, we have

E [Z] \leq \frac{2}{p} \sqrt{\frac{2}{n p}} (\sqrt{\frac{3}{} 2 ln (\frac{n p}{2})} + 1) .

Proof of Theorem 2.

First, we bound $V_{3, n}$ in terms of $V_{1}$ . From Lemma 1, $E [\frac{I {n (a) > 0}}{{ˆ π}_{D} (a)} - \frac{1}{π_{D} (a)}] \leq \frac{4}{π_{D} (a)}$ , while if $n π_{D}^{*} \geq 34$ , $E [\frac{I {n (a) > 0}}{{ˆ π}_{D} (a)} - \frac{1}{π_{D} (a)}] \leq \frac{2}{π_{D} (a)} \sqrt{\frac{2}{n π_{D} (a)}} (\sqrt{} \frac{3}{2} ln (\frac{n π_{D} (a)}{2}) + 1)$ . Plugging these into the definition of $V_{3, n}$ , we have $V_{3, n} \leq 4 V_{1}$ for all $n$ . Furthermore, when $n π_{D}^{*} \geq 34$ , thanks to monotonicity of the function $t \mapsto \sqrt{\frac{2}{t}} (\sqrt{\frac{3}{2} ln t} + 1)$ for $t > 1$ , we have

V_{3, n} \leq 2 V_{1} \sqrt{\frac{2}{n π_{D} *}} (\sqrt{\frac{3}{} 2 ln (\frac{n π_{D}^{*}}{2})} + 1) .

(5)

Now, to bound $V_{0, n} = {(\sum_{a} π (a) r_{Φ} (a) p_{a, n})}_{a}^{2} + \sum_{a} π^{2} (a) r_{Φ}^{2} (a) p_{a, n} (1 - p_{a, n})$ , remember that one lower bound for $R_{n}^{*}$ is $R_{max}^{2} {max}_{a} π^{2} (a) p_{a, n} / 4$ , where $R_{max}$ is the range for $r_{Φ}$ . ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: The range has to be in the final result.. Hence,

$V_{0, n}$	$= K^{2} {(\frac{1}{K} \sum a π (a) r_{Φ} (a) p_{a, n})}_{Φ}^{2} + \sum a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} (1 - p_{a, n})$
	$\leq K \sum a π^{2} (a) r_{Φ}^{2} (a) p_{a, n}^{2} + \sum a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} (1 - p_{a, n})$
	$\leq K \sum a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} \leq K^{2} max a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} .$	(6)

Hence, using $R_{n}^{*} \geq V_{1} / n$ , ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: constant?

M S E ({ˆ v}_{R e g})

\leq V_{0, n} + \frac{V_{1} + V_{3}}{n} \leq 4 K^{2} max a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} + 5 \frac{V_{1}}{n} \leq (4 K^{2} + 5) R_{n}^{*} .

(7)

On the other hand, assuming that ${min}_{a} σ^{2} (a) > 0$ , we also have

V_{0, n}

\leq K \sum a π^{2} (a) r_{Φ}^{2} (a) p_{a, n} \leq K max b \in A (\frac{r_{Φ}^{2} (b)}{σ^{2} (b)}) \sum a p_{a, n} π^{2} (a) σ^{2} (a) \leq K max b \in A (\frac{r_{Φ}^{2} (b)}{σ^{2} (b)}) \frac{V_{1}}{n},

where in the last inequality we used that $p_{a, n} \leq e^{- n π_{D} (a)}$ and $e^{- x} \leq 1 / x$ , which is true for any $x > 0$ , and finally also the definition of $V_{1}$ . Similarly to the previous case, we get

M S E ({ˆ v}_{R e g})

\leq {K max b \in A (\frac{r_{Φ}^{2} (b)}{σ^{2} (b)}) + 5} \frac{V_{1}}{n} \leq {K max b \in A (\frac{r_{Φ}^{2} (b)}{σ^{2} (b)} + 5)} R_{n}^{*} .

Combining this with (7) gives (4).

For the second part of the result, choose $π (a) = π_{D} (a) = 1 / K$ , $r_{Φ} (a) = 1$ . For $K \geq 2$ , $p_{a, n} = (1 - 1 / K)^{n} = e^{- n log (1 / (1 - 1 / K))} = e^{- n log (1 + 1 / (K - 1))} \geq e^{- n / (K - 1)}$ . Hence, we have $M S E ({ˆ v}_{R e g}) \geq (E [{ˆ v}_{R e g} - v_{Φ}^{π}])^{2} = {(\sum_{a} π (a) r_{Φ} (a) p_{a, n})}_{a}^{2} \geq e^{- 2 n / (K - 1)}$ . Now, consider the LR estimator. Choosing $σ^{2} = 0$ , we have $V_{1} = 0$ and so by creftype 1,

sup Φ : 0 \leq r_{Φ} \leq 1, σ_{Φ}^{2} = 0 M S E ({ˆ v}_{L R}) = sup Φ : 0 \leq r_{Φ} \leq 1, σ_{Φ}^{2} = 0 V_{2} / n \leq \frac{1}{n} .

Hence, $\frac{M S E ({ˆ v}_{R e g})}{R_{n}^{*} (π, π_{D}, 1, 0)} \geq \frac{e^{- 2 n / (K - 1)}}{{sup}_{Φ : 0 \leq r_{Φ} \leq 1, σ_{Φ}^{2} = 0} M S E ({ˆ v}_{L R})} \geq n e^{- 2 n / (K - 1)}$ .

Finally, the for the last part, fix any $π, π_{D}$ , $σ^{2}$ , $Φ$ such that $σ_{Φ}^{2} \leq σ^{2}$ . Then, for $n$ large enough, $M S E ({ˆ v}_{R e g}) \leq V_{0, n} + \frac{V_{1} + V_{3}}{n} \leq C e^{- n / C} + \frac{V_{1}}{n} (1 + C \sqrt{\frac{ln n}{n}})$ , where $C > 0$ is a problem dependent constant, and the second inequality used (5) and (6). Combining this with (1) of Theorem 1 gives the desired result. ∎

2.4 Simulation Results

Figure 1: nMSE of estimators against sample size.

This subsection corroborates our analysis with simulation results that empirically demonstrate the impact of key quantities on the MSE of the two estimators. Two sets of experiments are done, corresponding to the left and right panels in Figure 1. In all experiments, we repeat the data-generation process (with $π_{D}$ ) 10,000 times, and compute the MSE of each estimator. All reward distributions are normal distributions with $σ^{2} = 0.01$ and different means. We then plot normalized MSE (MSE multiplied by sample size $n$ ), or nMSE, against $n$ .

The first experiment is to compare the finite-time as well as asymptotic accuracy of ${ˆ v}_{L R}$ and ${ˆ v}_{R e g}$ . We choose $K = 10$ , $r_{Φ} (a) = a / K$ , $π (a) \propto a$ . Three choices of $π_{D}$ are used: (a) $π_{D} (a) \propto a$ , (b) $π_{D} (a) = 1 / K$ , and (c) $π_{D} (a) \propto (K - a)$ . These choices lead to increasing values of $V_{2}$ (with $V_{1}$ approximately fixed). Clearly, the nMSE of ${ˆ v}_{L R}$ remains constant, equal to $V_{1} + V_{2}$ , as predicted in Proposition 1. In contrast, the nMSE of ${ˆ v}_{R e g}$ is large when $n$ is small, because of the high bias, and then quickly converges to the asymptotic minimax rate $V_{1}$ (Theorem 2, part iii). As $V_{2}$ can be arbitrarily larger than $V_{1}$ , it follows that ${ˆ v}_{R e g}$ is preferred over ${ˆ v}_{L R}$ , as least for sufficiently large $n$ that is needed to drive the bias down. It should be noted that in practice, after $D^{n}$ is generated, it is easy to quantify the bias of ${ˆ v}_{R e g}$ simply by identifying the set of actions $a$ with $n (a) = 0$ .

The second experiment is to show how $K$ affects the nMSE of ${ˆ v}_{R e g}$ . Here, we choose $π_{D} = 1 / K$ , $r_{Φ} (a) = a / K$ , $π (a) \propto a$ , and vary $K \in {50, 100, 200, 500, 1000}$ . As Figure 1 (right) shows, a larger $K$ gives ${ˆ v}_{R e g}$ a harder time, which is consistent with Theorem 2 (part i). Not only does the maximum nMSE grow approximately linearly with $K$ , the number of samples needed for nMSE to start decreasing also scales roughly as $(K - 1) / 2$ , as indicated by part ii of Theorem 2. ^{size=,color=red!20!white,}^{size=,color=red!20!white,}todo: size=,color=red!20!white,Lihong: Why $(K - 1) / 2$ instead of $K / 2$ ?

3 Extensions

In this section, we consider extensions of our previous results to contextual bandits and Markovian Decision Processes, while implications to semi-supervised learning (Zhu and Goldberg, 2009) are discussed in the supplementary material.

3.1 Contextual Bandits

The problem setup is as follows: In addition to the finite action set $A = {1, 2, \dots, K}$ , we are also given a context set $X = {1, 2, \dots, M}$ . A policy now is a map $π : X \to [0, 1]^{A}$ such that for any $x \in X$ , $π (x)$

is a probability distribution over the action space

A

. For notational convenience, we will use

π (a | x)

instead of

π (x) (a)

. The set of policies over

X

and

A

will be denoted by

Π (X, A)

. The process generating the data

D^{n} = {(X_{i}, A_{i}, R_{i})}_{1 \leq i \leq n}

is described by the following:

(X_{i}, A_{i}, R_{i})

are independent copies of

(X, A, R)

, where

X \sim μ (\cdot)

A \sim π_{D} (\cdot | X)

and

R \sim Φ (\cdot | A, X)

for some unknown family of distributions

{Φ (\cdot | a, x)}_{a \in A, x \in X}

and known policy

π_{D} \in Π (X, A)

and context distribution

μ

. For simplicity, we fix

R_{m a x} = 1

We are also given a known target policy $π \in Π (X, A)$ and want to estimate its value, $v_{Φ}^{π, μ} := E_{X \sim μ, A \sim π (\cdot | X), R \sim Φ (\cdot | A, X)} [R]$ based on the knowledge of $D^{n}$ , $π_{D}$ , $μ$ and $π$ , where the quality of an estimate $ˆ v$ constructed based on $D^{n}$ (and $π, π_{D}, μ$ ) is measured by its mean squared error, $M S E (ˆ v) := E [(ˆ v - v_{Φ}^{π, μ})^{2}]$ , just like in the case of contextless bandits. ^{size=,color=green!20!white,}^{size=,color=green!20!white,}todo: size=,color=green!20!white,Csaba: mean squared error, or mean-squared error? in any ways, write it one way. Let $σ_{Φ}^{2} (x, a) = V (R)$ for $R \sim Φ (\cdot | x, a)$ , $x \in X, a \in A$ . An estimator $A$ can be considered as a function that maps $(μ, π, π_{D}, D^{n})$ to an estimate of $v_{Φ}^{π, μ}$ , denoted ${ˆ v}_{A} (μ, π, π_{D}, D^{n})$ . Fix $σ^{2} := (σ^{2} (x, a))_{x \in X, a \in A}$ . The minimax optimal risk subject to $σ_{Φ}^{2} (x, a) \leq σ^{2} (x, a)$ for all $x \in X, a \in A$ is defined by $R_{n}^{*} (μ, π, π_{D}, σ^{2}) := {inf}_{A} {sup}_{Φ : σ_{Φ}^{2} \leq σ^{2}} E [({ˆ v}_{A} (μ, π, π_{D}, D^{n}) - v_{Φ}^{π, μ})^{2}] .$

The main observation is that the estimation problem for the contextual case can actually be reduced to the contextless bandit case by treating the context-action pairs as “actions” belonging to the product space $X \times A$ . For any policy $π$ , by slightly abusing notation, let $(μ \otimes π) (x, a) = μ (x) π (a | x)$

be the joint distribution of

(X, A)

when

X \sim μ (\cdot)

A \sim π (\cdot | X)

. This way, we can map any contextual policy evaluation problem defined by

μ

π_{D}

π

Φ

and a sample size

n

into a contextless policy evaluation problem defined by

μ \otimes π_{D}

μ \otimes π

Φ

with action set

X \times A

. Therefore, with

V_{1}

and

V_{2}

defined similarly, one can conclude the following results:

Theorem 3.

Pick any $n > 0$ , $μ$ , $π_{D}$ , $π$ and $σ^{2}$ . Then, one has

On Minimax Optimal Offline Policy Evaluation

MOTS: Minimax Optimal Thompson Sampling

Simulation Based Algorithms for Markov Decision Processes and Multi-Action Restless Bandits

Stateful Offline Contextual Policy Evaluation and Learning

Markov Decision Processes under Ambiguity

VOI-aware MCTS

Off-Policy Evaluation in Partially Observed Markov Decision Processes

Minimax Off-Policy Evaluation for Multi-Armed Bandits

1 Introduction

2 Multi-armed Bandit

2.1 A Minimax Lower Bound

Theorem 1.

Proof.

Corollary 1.

Proof.

2.2 Likelihood Ratio Estimator

Proposition 1.

2.3 Regression Estimator

Proposition 2.

Proof sketch.

Theorem 2 (Minimax Optimality of the Regression Estimator).

Lemma 1.

Proof of Theorem 2.

2.4 Simulation Results

3 Extensions

3.1 Contextual Bandits

Theorem 3.

On Minimax Optimal Offline Policy Evaluation

Related Research

MOTS: Minimax Optimal Thompson Sampling

Simulation Based Algorithms for Markov Decision Processes and Multi-Action Restless Bandits

Stateful Offline Contextual Policy Evaluation and Learning

Markov Decision Processes under Ambiguity

VOI-aware MCTS

Off-Policy Evaluation in Partially Observed Markov Decision Processes

Minimax Off-Policy Evaluation for Multi-Armed Bandits

1 Introduction

2 Multi-armed Bandit

2.1 A Minimax Lower Bound

Theorem 1.

Proof.

Corollary 1.

Proof.

2.2 Likelihood Ratio Estimator

Proposition 1.

2.3 Regression Estimator

Proposition 2.

Proof sketch.

Theorem 2 (Minimax Optimality of the Regression Estimator).

Lemma 1.

Proof of Theorem 2.

2.4 Simulation Results

3 Extensions

3.1 Contextual Bandits

Theorem 3.