Online Variance Reduction for Stochastic Optimization

1 Introduction

Empirical risk minimization (ERM) is among the most important paradigms in machine learning, and is often the strategy of choice due to its generality and statistical efficiency. In ERM, we draw a set of samples

D = {x_{1}, \dots, x_{n}} \subset X

from the underlying data distribution and we aim to find a solution

w \in W

that minimizes the empirical risk,

min w \in W L (w) := \frac{1}{n} n \sum i = 1 ℓ (x_{i}, w),

(1)

where $ℓ : X \times W \to R$

is a given loss function, and

W \subseteq R^{d}

is usually a compact domain.

In this work we are interested in sequential procedures for minimizing the ERM objective, and relate to such methods as ERM solvers. More concretely, we focus on the regime where the number of samples $n$

is very large, and it is therefore desirable to employ ERM solvers that only require few passes over the dataset. There exists a rich arsenal of such efficient solvers which have been investigated throughout the years, with the canonical example from this category being Stochastic Gradient Descent (SGD).

Typically, such methods require an unbiased estimate of the loss function at each round, which is usually generated by sampling a few points uniformly at random from the dataset. However, by employing uniform sampling, these methods are insensitive to the intrinsic structure of the data. In case of SGD, for example, some data points might produce large gradients, but they are nevertheless assigned the same probability of being sampled as any other point. This ignorance often results in high-variance estimates, which is likely to degrade the performance.

The above issue can be mended by employing non-uniform importance sampling. And indeed, we have recently witnessed several techniques to do so: Zhao and Zhang (2015) and similarly Needell et al. (2014), suggest using prior knowledge on the gradients of each datapoint in order to devise predefined importance sampling distributions. Stich et al. (2017) devise adaptive sampling techniques guided by a robust optimization approach. These are only a few examples of a larger body of work (Bouchard et al., 2015; Alain et al., 2015; Csiba and Richtárik, 2016).

Interestingly, the recent works of Namkoong et al. (2017) and Salehi et al. (2017) formulate the task of devising importance sampling distributions as an online learning problem with bandit feedback. In this context, they think of the algorithm, which adaptively chooses the distribution, as a player that competes against the ERM solver. The goal of the player is to minimize the cumulative variance of the resulting (gradient) estimates. Curiously, both methods rely on some form of the “linearization trick”¹¹1 By “linearization trick” we mean that these methods update according to a first order approximation of the costs rather than the costs themselves. to resort to the analysis of the EXP3 (Auer et al., 2002).

On the other hand, the theoretical guarantees of the above methods are somewhat limited. Strictly speaking, none of them provides regret guarantees with respect to the best fixed distribution in hindsight: Namkoong et al. (2017) only compete with the best distribution among a subset

of the simplex (around the uniform distribution). Conversely,

Salehi et al. (2017) compete against a solution which might perform worse than the best in hindsight up to a multiplicative factor of

3

In this work, we adopt the above mentioned online learning formulation, and design novel importance sampling techniques. Our adaptive sampling procedure is simple and efficient, and in contrast to previous work, we are able to provide regret guarantees with respect to the best fixed point among the simplex. As our contribution, we

motivate theoretically why regret minimization is meaningful in this setting,
propose a novel bandit algorithm for variance reduction ensuring regret of $~ O (n^{1 / 3} T^{2 / 3})$ ,
empirically validate our method and provide an efficient implementation²²2The source code is available at https://github.com/zalanborsos/online-variance-reduction.

On the technical side, we do not rely on a “linearization trick” but rather directly employ a scheme based on the classical Follow-the-Regularized-Leader approach. Our analysis entails several technical challenges, most notably handling unbounded cost functions while only receiving partial (bandit) feedback. Our design and analysis draws inspiration from the seminal works of Auer et al. (2002) and Abernethy et al. (2008). Although we present our method for choosing datapoints, it naturally applies to choosing coordinates in coordinate descent or even blocks of thereof (Allen-Zhu et al., 2016; Perekrestenko et al., 2017; Nesterov, 2012; Necoara et al., 2011). More broadly, the proposed algorithm can be incorporated in any sequential algorithm that relies on an unbiased estimation of the loss. A prominent application of our method is variance reduction for SGD, which can be achieved by considering gradient norms as losses, i.e., replacing $ℓ (w, x_{i}) \leftrightarrow ∥ \nabla ℓ (w, x_{i}) ∥$ . With this modification, our method is minimizing the cumulative variance of the gradients throughout the optimization process. The latter quantity directly affects the quality of optimization (we elaborate on this in Appendix A).

The paper is organized as follows. In Section 2, we formalize the online learning setup of variance reduction and motivate why regret is a suitable performance measure. As the first step of our analysis, we investigate the full information setting in Section 3, which serves as a mean for studying the bandit setting in Section 4. Finally, we validate our method empirically and provide the detailed discussion of the results in Section 5.

2 Motivation and Problem Definition

Typical sequential solvers for ERM usually require a fresh unbiased estimate ${~ L}_{t}$ of the loss $L_{t}$ at each round, which is obtained by repeatedly sampling from the dataset. The template of Figure 1 captures a rich family of such solvers such as SGD, SAGA (Defazio et al., 2014), SVRG (Johnson and Zhang, 2013), and online $k$ -Means (Bottou and Bengio, 1995).

Sequential Optimization Procedure for ERM

Figure 1: Template of a sequential procedure for minimizing the ERM objective. At each round, we devise a fresh unbiased estimate

{~ L}_{t} (\cdot)

of the empirical loss, then we update the solution based on the previous solution

w_{t}

and

{~ L}_{t} (\cdot)

A natural way to devise the unbiased estimates ${~ L}_{t}$ is to sample $i_{t} \in {1, \dots, n}$ uniformly at random and return ${~ L}_{t} (w) = ℓ (x_{i_{t}}, w)$ . Indeed, uniform sampling is the common practice when applying SGD, SAGA, SVRG and online $k$ -Means. Nevertheless, any distribution $p$ in the probability simplex $Δ$ induces an unbiased estimate. Concretely, sampling an index $i \sim p$ induces the estimate

~ L (w) := \frac{1}{n \cdot p (i)} \cdot ℓ (x_{i}, w)

(2)

and it is immediate to show that $E_{x_{i} \sim p} [~ L (w)] = L (w)$ . This work is concerned with efficient ways of choosing a “good” sequence of sampling distributions ${p_{1} (\cdot), \dots, p_{T} (\cdot)}$ .

It is well known that the performance of typical solvers (e.g. SGD, SAGA, SVRG) improves as the variance of the estimates ${~ L}_{t} (w_{t})$ is becoming smaller. Thus, a natural criterion for measuring the performance of a sampling distribution $p$ is the variance of the induced estimate

{Var}_{p} (~ L (w)) = \frac{1}{n^{2}} n \sum i = 1 \frac{ℓ^{2} (x_{i}, w)}{p (i)} - L^{2} (w) .

Denoting $ℓ_{t} (i) := ℓ (x_{i}, w_{t})$ and noting that the second term is independent of $p$ , we may now cast the task of sequentially choosing the sampling distributions as the online optimization problem shown in Figure 2

. In this protocol, we treat the sequential solver as an adversary that chooses a sequence of loss vectors

{ℓ_{t}}_{t \in [T]} \subset R^{n}

, where

t \in [T]

denotes

t \in {1, \dots, T}

. Each loss vector is a function of

w_{t}

, the solution chosen by the solver in the corresponding round (note that we abstract out this dependence of

ℓ_{t}

w_{t}

). The cost³³3We use the term “cost function” to refer to

f

in order to distinguish it from the loss

ℓ

\frac{1}{n^{2}} f_{t} (p_{t})

that the player incurs at round

t

is the second moment of the loss estimate, which is induced by the distribution chosen by the player at round

t

Online Variance Reduction Protocol

Figure 2: Online variance reduction protocol with bandit feedback

Next, we define the regret, which is our performance measure for the player,

{Regret}_{T} = \frac{1}{n^{2}} (T \sum t = 1 f_{t} (p_{t}) - min p \in Δ T \sum t = 1 f_{t} (p)) .

Our goal is to devise a no-regret algorithm such that ${lim}_{T \to \infty} {Regret}_{T} / T = 0$ , which in turn guarantees that we recover asymptotically the best fixed sampling distribution. In the bandit feedback setting, the player aims to minimize its expected regret $E [{Regret}_{T}]$ , where the expectation is taken with respect to the randomized choices of the player and the adversary. Note that we allow the choices of the adversary to depend on the past choices of the player.

There are few noteworthy comments regarding the above setup. First, it is immediate to verify that the cost functions $f_{1}, \dots, f_{T}$ are convex in $Δ$ , therefore this is an online convex optimization problem. Secondly, the cost functions are unbounded in $Δ$ , which poses a challenge in ensuring no-regret. Finally, notice that the player receives a bandit feedback, i.e., he is allowed to inspect the losses only at the coordinate $I_{t}$ chosen at time $t$ . To the best of our knowledge, this is the first natural setting where, as we will show, it is possible to provide no regret guarantees despite bandit feedback and unbounded costs.

Throughout this work, we assume that the losses are bounded, $l_{t}^{2} (i) \leq L$ for all $i \in [n]$ and $t \in [T]$ . Note that our analysis may be extended to the case where the bounds are instance-dependent, i.e., $l_{t}^{2} (i) \leq L_{i}$ for all $i \in [n]$ and $t \in [T]$ . In practice, it can be beneficial to take into account the different $L_{i}$ ’s, as we demonstrate in our experiments.

2.1 Is Regret a Meaningful Performance Measure?

Let us focus on the family of ERM solvers depicted in Figure 1. As discussed above, devising loss estimates such that ${~ L}_{t} (w_{t})$ has low variance is beneficial for such solvers — in case of SGD, this is due to strong connection between the cumulative variance of gradients and the quality of optimization that we discuss in more detail in Appendix A. Translating this observation into the online variance reduction setting suggests a natural performance measure: rather than competing with the best fixed distribution in hindsight, we would like to compete against the sequence of best distributions per-round ${p_{t}^{*} \leftarrow a r g m i n \sum_{i = 1}^{n} ℓ_{t}^{2} (i) / p (i)}_{t \in [T]}$ . This optimal sequence ensures zero variance in every round, and is therefore the ideal baseline to compete against. This also raises the question whether regret guarantees, which compare against the best fixed distribution in hindsight, are at all meaningful in this context. Note that regret minimization is meaningful in stochastic optimization, when we assume that the losses are generated i.i.d. from some fixed distribution (Cesa-Bianchi et al., 2004). Yet, this certainly does not apply in our case since losses are non-stationary and non-oblivious.

Unfortunately, ensuring guarantees compared to the sequence of best distributions per-round seems generally hard. However, as we show next, regret is still a meaningful measure for sequential ERM solvers. Concretely, recall that our ultimate goal is to minimize the ERM objective. Thus, we are only interested in ERM solvers that actually converge to a (hopefully good) solution for the ERM problem. More formally, let us define $ℓ_{*} (i)$ as follows,

ℓ_{*} (i) := lim t \to \infty ℓ_{t} (i),

where we recall that $ℓ_{t} (i) := ℓ (x_{i}, w_{t})$ , and assume the above limit to exist for every $i \in [n]$ . We will also denote $L_{*} := \frac{1}{n} \sum_{i = 1}^{n} ℓ_{*} (i)$ . Moreover, let us assume that the asymptotic solution is better on average than any of the sequential solutions in the following sense,

\frac{1}{T} T \sum t = 1 L (w_{t}) \geq L_{*}, \forall T \geq 1

where $L (w_{t}) := \frac{1}{n} \sum_{i = 1}^{n} ℓ (x_{i}, w_{t})$ . This assumption naturally holds when the ERM solver converges to the optimal solution for the problem, which applies for SGD in the convex case.

The next lemma shows that under these mild assumptions, competing against the best fixed distribution in hindsight is not far from competing against the ideal baseline.

Lemma 1.

Consider the online variance reduction setting, and for any $i \in [n]$ denote $V_{T} (i) = \sum_{t = 1}^{T} (ℓ_{t} (i) - ℓ_{*} (i))^{2}$ . Assuming that the losses, $l_{t} (i)$ , are non-negative for all $i \in [n]$ , $t \in [T]$ , the following holds for any $T \geq 1$ ,

\frac{1}{n^{2}} min p \in Δ T \sum t = 1 f_{t} (p) \leq \frac{1}{n^{2}} T \sum t = 1 min p \in Δ f_{t} (p) + 2 \sqrt{T} L_{*} \cdot \frac{}{1} n n \sum i = 1 \sqrt{V_{T} (i)} + {(\frac{1}{n} n \sum i = 1 \sqrt{V_{T} (i)})}^{2} .

Thus, the above lemma connects the convergence rate of the ERM solver to the benefit that we get by regret minimization. It shows that the benefit is larger if the ERM solver converges faster. As an example, let us assume that $| ℓ_{t} (i) - ℓ_{*} (i) | \leq O (1 / \sqrt{t})$ , which loosely speaking holds for SGD. This assumption implies $V_{T} (i) \leq O (log (T))$ , hence by Lemma 1 the regret guarantees translate into guarantees with respect to the ideal baseline, with an additional cost of $~ O (\sqrt{T})$ .

3 Full Information Setting

In this section, we analyze variance reduction with full-information feedback. We henceforth consider the same setting as in Figure 2, with the difference that in each round the player receives as a feedback the loss vector at all points $(l_{t} (1), l_{t} (2), \dots, l_{t} (n))$ instead of only $l_{t} (I_{t})$ . We introduce a new algorithm based on the FTRL approach, and establish an $O (\sqrt{T})$ regret bound for our method in Theorem 3. While this setup in itself has little practical relevance, it later serves as a mean for investigating the bandit setting.

Follow-the-Regularized-Leader (FTRL) is a powerful approach to online learning problems. According to FTRL, in each round, one selects a point that minimizes the cost functions over past rounds plus a regularization term, i.e., $p_{t} \leftarrow {a r g m i n}_{p \in Δ} \sum_{τ = 1}^{t - 1} f_{τ} (p) + R (p)$ . The regularizer $R$ usually assures that the choices do not change abruptly over the rounds. We choose $R (p) = γ \sum_{i = 1}^{n} \frac{1}{p (i)}$ which allows to write FTRL as

p_{t} \leftarrow a r g m i n p \in Δ t - 1 \sum τ = 1 f_{τ} (p) + γ n \sum i = 1 \frac{1}{p (i)} .

(3)

The regularizer $R (p) = γ \sum_{i = 1}^{n} 1 / p (i)$ is a natural candidate in our setting, since it has the same structural form as the cost functions. It also prevents FTRL from assigning vanishing probability mass to any component, thus ensuring that the incurred costs never explode. Moreover, $R$ assures a closed form solution to the FTRL as the following lemma shows.

Lemma 2.

Denote $l_{1 : t}^{2} (i) := \sum_{τ = 1}^{t} ℓ_{τ}^{2} (i)$ . The solution to Eq. (3) is $p_{t} (i) \propto \sqrt{ℓ_{1 : t - 1}^{2} (i) + γ}$ .

Proof sketch.

Recalling $f_{t} (p) = \sum_{i = 1}^{n} \frac{ℓ_{t}^{2} (i)}{p (i)}$ , allows to write the FTRL objective as follows,

t - 1 \sum τ = 1 f_{τ} (p) + γ n \sum i = 1 1 / p (i) = n \sum i = 1 (ℓ_{1 : t - 1}^{2} (i) + γ) / p (i) .

It is immediate to validate that the offered solution satisfies the first order optimality conditions in $Δ$ . Global optimality follows since the FTRL objective is convex in the simplex.

∎

We are interested in the regret incurred by our method. The following theorem shows that, despite the non-standard form of the cost functions, we can obtain $O (\sqrt{T})$ regret.

Theorem 3.

Setting $γ = L$ , the regret of the FTRL scheme proposed in Equation (3) is:

{Regret}_{T} \leq \frac{27 \sqrt{L}}{n} (n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)}) + 44 L .

Furthermore, since $ℓ_{t}^{2} (i) \leq L$ we have ${Regret}_{T} \leq 27 L \sqrt{T} + 44 L$ .

Before presenting the proof, we briefly describe it. Trying to apply the classical FTRL regret bounds, we encounter a difficulty, namely that the regularizer in Equation (3) can be unbounded. To overcome this issue, we first consider competing with the optimal distribution on a restricted simplex where $R (\cdot)$ is bounded. Then we investigate the cost of considering the restricted simplex instead of the full simplex.

Along the lines described above, consider the simplex $Δ$ and the restricted simplex $Δ^{'} = {p \in Δ | p (i) \geq p_{min}, \forall i \in [n]}$ where $p_{min} \leq 1 / n$ is to be defined later. We can now decompose the regret as follows,

n^{2} \cdot {Regret}_{T} = T \sum t = 1 f_{t} (p_{t}) - min p \in Δ^{'} T \sum t = 1 f_{t} (p)      (A) + min p \in Δ^{'} T \sum t = 1 f_{t} (p) - min p \in Δ T \sum t = 1 f_{t} (p)      (B) .

(4)

We continue by separately bounding the above terms. To bound $(A)$ , we will use standard tools which relate the regret to the stability of the FTRL decision sequence (FTL-BTL lemma). Term $(B)$ is bounded by a direct calculation of the minimal values in $Δ$ and $Δ^{'}$ .

The following lemma bounds term $(A)$ .

Lemma 4.

Setting $γ = L$ , we have:

T \sum t = 1 f_{t} (p_{t}) - min p \in Δ^{'} T \sum t = 1 f_{t} (p) \leq 22 n \sqrt{L} \cdot (n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)}) + 22 n^{2} L + \frac{n L}{p_{min}} .

Proof sketch of Lemma 4 .

The regret of FTRL may be related to the stability of the online decision sequence as shown in the following lemma due to Kalai and Vempala (2005) (proof can also be found in Hazan (2011) or in Shalev-Shwartz et al. (2012)):

Lemma 5.

Let $K$ be a convex set and $R : K \mapsto R$ be a regularizer. Given a sequence of cost functions ${f_{t}}_{t \in [T]}$ defined over $K$ , then setting $p_{t} = {a r g m i n}_{p \in Δ} \sum_{τ = 1}^{t - 1} f_{τ} (p) + R (p)$ ensures,

T \sum t = 1 f_{t} (p_{t}) - T \sum t = 1 f_{t} (p) \leq T \sum t = 1 (f_{t} (p_{t}) - f_{t} (p_{t + 1})) + (R (p) - R (p_{1})), \forall p \in K

Notice that $R (p) = L \sum_{i = 1}^{n} 1 / p (i)$ is non-negative and bounded by $n L / p_{min}$ over $Δ^{'}$ . Thus, applying the above lemma implies that $\forall p \in Δ^{'}$ ,

T \sum t = 1 f_{t} (p_{t}) - T \sum t = 1 f_{t} (p) \leq T \sum t = 1 (f_{t} (p_{t}) - f_{t} (p_{t + 1})) + \frac{n L}{p_{min}} \leq T \sum t = 1 n \sum i = 1 ℓ_{t}^{2} (i) (\frac{1}{p_{t} (i)} - \frac{1}{p_{t + 1} (i)}) + \frac{n L}{p_{min}} .

Using the closed form solution for the $p_{t}$ ’s (see Lemma. 2) enables us to upper bound the last term as follows,

T \sum t = 1 n \sum i = 1 ℓ_{t}^{2} (i) (\frac{1}{p_{t} (i)} - \frac{1}{p_{t + 1} (i)})

\leq 22 n \sqrt{L} n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i) + L} .

(5)

Combining the above with $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ completes the proof.

∎

The next lemma bounds term $(B)$ .

Lemma 6.

min p \in Δ^{'} T \sum t = 1 f_{t} (p) - min p \in Δ T \sum t = 1 f_{t} (p) \leq 6 n \cdot p_{min} \cdot {(n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)})}^{2}

Proof sketch of Lemma 6.

Using first order optimality conditions we are able show that the minimal value of the $\sum_{t = 1}^{T} f_{t} (p)$ over $Δ$ is exactly ${(\sum_{i = 1}^{n} \sqrt{ℓ_{1 : t}^{2} (i)})}^{2}$ . Similar analysis allows to extract a closed form solution to the best in hindsight over $Δ^{'}$ . This in turn enables to upper bound the minimal value over $Δ^{'}$ by ${(\sum_{i = 1}^{n} \sqrt{ℓ_{1 : t}^{2} (i)})}^{2} / {(1 - n \cdot p_{min})}_{min}^{2}$ . Combining these bounds together with $p_{min} \leq 1 / 2 n$ we are able to prove the lemma.

∎

Proof of Theorem 3.

Combining Lemma 4 and 6, we have after dividing by $n^{2}$ ,

{Regret}_{T} \leq \frac{22 \sqrt{L}}{n} \cdot (n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)}) + 22 L + \frac{L}{n \cdot p_{min}} + \frac{6 \cdot p_{% min}}{n} \cdot {(n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)})}^{2}

Since the choice of $p_{min}$ is arbitrary and is relevant only for the theoretical analysis, we can set it to $p_{min} = min {1 / (2 n), \sqrt{L} / (\sqrt{6} \sum_{i = 1}^{n} \sqrt{ℓ_{1 : T}^{2} (i)})}$ that yields the final result. ∎

4 The Bandit Setting

In this section, we investigate the bandit setting (see Figure 2) which is of great practical appeal as we described in Section 2. Our method for the bandit setting is depicted in Algorithm 1, and it ensures a bound of $~ O (n^{1 / 3} T^{2 / 3})$ on the expected regret (see Theorem 8). Importantly, this bound holds even for non-oblivious adversaries. The design and analysis of our method builds on some of the ideas that appeared in the seminal work of Auer et al. (2002).

Algorithm 1 is using the bandit feedback in order to design an unbiased estimate of the true loss $(ℓ_{t} (1), \dots, ℓ_{t} (n))$ in each round. These estimates are then used instead of the true losses by the full information FTRL algorithm that was analyzed in the previous section. We do not directly play according to the FTRL predictions but rather mix them with a uniform distribution. Mixing is necessary in order to ensure that the loss estimates are bounded, which is a crucial condition used in the analysis. Next we elaborate on our method and its analysis.

The algorithm samples⁴⁴4The sampling and update in the presented form have a complexity of $O (n)$ . There is a standard way to improve this based on segment trees that gives $O (log n)$ for sampling and update. A detailed description of this idea can be found in section A.4. of Salehi et al. (2017). The efficient implementation of the sampler is available at https://github.com/zalanborsos/online-variance-reduction an arm $I_{t} \sim {~ p}_{t}$ at every round and receives a bandit feedback $ℓ_{t} (I_{t})$ . This may be used in order to construct an estimate of the true (squared) loss as follows,

and it is immediate to validate that the above is unbiased in the following sense,

E [{~ ℓ}_{t}^{2} (i) | {~ p}_{t}, ℓ_{t}] = ℓ_{t}^{2} (i), \forall i \in [n] .

Analogously to the previous section it is natural to define modified cost functions as

{~ f}_{t} (p) = n \sum i = 1 {~ ℓ}_{t}^{2} (i) / p (i) .

Clearly, ${~ f}_{t}$ is an unbiased estimate of the true cost, $E [{~ f}_{t} (p) | {~ p}_{t}, ℓ_{t}] = f_{t} (p)$ . From now on we omit the conditioning on ${~ p}_{t}, ℓ_{t}$ for notational brevity.

Having devised an unbiased estimate, we could return to the full information analysis of FTRL with the modified losses. However, this poses a difficulty, since the modified losses can possibly be unbounded. We remedy this by mixing the FTRL output, $p_{t}$ , with a uniform distribution. Mixing encourages exploration, and in turn gives a handle on the possibly unbounded modified losses. Let $θ \in [0, 1]$ , and define

{~ p}_{t} (i) = (1 - θ) \cdot p_{t} (i) + θ / n .

Indeed, since ${~ p}_{t} (i) \geq θ / n$ , we have ${~ ℓ}_{t}^{2} (i) \leq n L / θ$ .

Input:

θ

L

n

Initialize

w (i) = 0

for all

i \in [n] .

for

t = 1

T

p_{t} (i) \propto \sqrt{w (i) + L \cdot n / θ}

{~ p}_{t} (i) = (1 - θ) \cdot p_{t} (i) + θ / n

, for all

i \in [n]

Draw

I_{t} \sim {~ p}_{t}

and play

I_{t}

Receive feedback

l_{t} (I_{t})

, and update

w (I_{t}) \leftarrow w (I_{t}) + l_{t}^{2} (I_{t}) / {~ p}_{t} (I_{t})

end for

Algorithm 1 Variance Reducer Bandit (VRB)

We start with analyzing the pseudo-regret of our algorithm, where we compare the cost incurred by the algorithm to the cost incurred by the optimal distribution in expectation. The pseudo-regret is defined below,

\frac{1}{n^{2}} min p \in Δ E [T \sum t = 1 f_{t} ({~ p}_{t}) - T \sum t = 1 f_{t} (p)],

(6)

where the expectation is taken with respect to both the player’s choices and the loss realizations. The pseudo-regret is only a lower bound for the expected regret, with an equality when the adversary is oblivious, i.e., does not take the past choices of the player into account.

Theorem 7.

Let $θ = (n / T)^{1 / 3}$ . Assuming $T \geq n$ , Algorithm 1 ensures the following bound,

\frac{1}{n^{2}} min p \in Δ E [T \sum t = 1 f_{t} ({~ p}_{t}) - T \sum t = 1 f_{t} (p)] \leq 74 L n^{\frac{1}{3}} T^{\frac{2}{3}} .

Proof sketch of Theorem 7.

Using the unbiasedness of the modified costs we have

min p \in Δ E [T \sum t = 1 f_{t} ({~ p}_{t}) - T \sum t = 1 f_{t} (p)] = min p \in Δ E [T \sum t = 1 {~ f}_{t} ({~ p}_{t}) - T \sum t = 1 {~ f}_{t} (p)] .

We can decompose $\frac{1}{n^{2}} {min}_{p \in Δ} E [\sum_{t = 1}^{T} {~ f}_{t} ({~ p}_{t}) - \sum_{t = 1}^{T} {~ f}_{t} (p)]$ into the following terms:

\frac{1}{n^{2}} E [T \sum t = 1 {~ f}_{t} ({~ p}_{t}) - T \sum t = 1 {~ f}_{t} (p_{t})]      (A) + \frac{1}{n^{2}} min p \in Δ E [T \sum t = 1 {~ f}_{t} (p_{t}) - T \sum t = 1 {~ f}_{t} (p)]      (B)

where $(A)$ is the cost we incur by mixing, and $(B)$ is upper bounded by the regret of playing FTRL with the modified losses. Now we inspect each term separately.

An upper bound of $θ L T$ on $(A)$ results from the following simple observation:

\frac{1}{{~ p}_{t} (i)} - \frac{1}{p_{t} (i)} \leq n θ .

For bounding $(B)$ , notice that $p_{t}$ is performing FTRL over the modified cost sequence. Combining this together the bound ${~ ℓ}_{t}^{2} (i) \leq n L / θ$ allows us to apply Theorem 3 and get,

\frac{1}{n^{2}} (T \sum t = 1 {~ f}_{t} (p_{t}) - min p \in Δ T \sum t = 1 {~ f}_{t} (p)) \leq 27 \sqrt{\frac{L}{n θ}} (n \sum i = 1 \sqrt{{~ ℓ}_{1 : T}^{2} (i)}) + \frac{44 n L}{θ} .

(7)

Due to Jensen’s inequality we have

E [n \sum i = 1 \sqrt{{~ ℓ}_{1 : T}^{2} (i)}] \leq n \sum i = 1 \sqrt{E [{~ ℓ}_{1 : T}^{2} (i)]} = n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)} .

Putting these results together, we get an upper bound on the pseudo-regret which we can optimize in terms of $θ$ :

\frac{1}{n^{2}} min p \in Δ E [T \sum t = 1 f_{t} ({~ p}_{t}) - T \sum t = 1 f_{t} (p)] \leq θ L T + 27 \sqrt{\frac{L}{n θ}} (n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)}) + \frac{44 n L}{θ} .

Using the bound $\sum_{i = 1}^{n} \sqrt{ℓ_{1 : T}^{2} (i)} \leq n \sqrt{L T}$ and since we assumed $T \geq n$ , we can set $θ = (n / T)^{1 / 3}$ to get the result. Note that $θ$ is dependent on knowing $T$ in advance. If we do not assume that this is possible, we can use the “doubling trick” starting from $T = n$ and incur an additional constant multiplier in the regret. ∎

Ultimately, we are interested in the expected regret, where we allow the adversary to make decisions by taking into account the player’s past choices, i.e., to be non-oblivious. Next we present the main result of this paper, which establishes a $~ O (n^{1 / 3} T^{2 / 3})$ regret bound, where the $~ O$ notation hides the logarithmic factors.

Theorem 8.

Assuming $T \geq n$ , the following holds for the expected regret,

\frac{1}{n^{2}} E [T \sum t = 1 f_{t} ({~ p}_{t}) - min p \in Δ T \sum t = 1 f_{t} (p)] \leq ~ O (L n^{\frac{1}{3}} T^{\frac{2}{3}}) .

Proof sketch of Theorem 8.

Using the unbiasedness of the modified costs allows to decompose the regret as follows,

$n^{2} E [{Regret}_{T}]$	$= E [T \sum t = 1 f_{t} ({~ p}_{t}) - min p \in Δ T \sum t = 1 f_{t} (p)]$
	$= E [T \sum t = 1 {~ f}_{t} ({~ p}_{t}) - min p \in Δ T \sum t = 1 {~ f}_{t} (p)] + E [min p \in Δ T \sum t = 1 {~ f}_{t} (p) - min p \in Δ T \sum t = 1 f_{t} (p)]$
	$\leq n^{2} O (L n^{1 / 3} T^{2 / 3}) + E ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ {(n \sum i = 1 \sqrt{{~ ℓ}_{1 : T}^{2} (i)})}^{2} - {(n \sum i = 1 \sqrt{ℓ_{1 : T}^{2} (i)})}^{2}      (A) ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦,$	(8)

where the last line uses Equation (7) together with Jensen’s inequality (similarly to the proof of Theorem 7). We have also used the closed form solution for the minimal values of $\sum_{t} f_{t} (p)$ and $\sum_{t} {~ f}_{t} (p)$ over the simplex.

Our approach to bounding the remaining term is to establish high probability bound for $(A)$ . In order to do so we shall bound the following differences ${~ ℓ}_{1 : T}^{2} (i) - ℓ_{1 : T}^{2} (i)$ . This can be done by applying the appropriate concentration results described below.
Bounding ${~ ℓ}_{1 : T}^{2} (i) - ℓ_{1 : T}^{2} (i)$ . Fix $i \in [n]$ and define $Z_{t, i} := {~ ℓ}_{t}^{2} (i) - ℓ_{t}^{2} (i)$ . Recalling that $E [{~ ℓ}_{t}^{2} (i) | {~ p}_{t}, ℓ_{t}] = ℓ_{t}^{2} (i)$ , we have that ${Z_{t, i}}_{t \in [T]}$ is a martingale difference sequence with respect to the filtration ${F_{t}}_{t \in [T]}$ associated with the history of the strategy. This allows us to apply a version of Freedman’s inequality (Freedman, 1975), which bounds the sum of differences with respect to their cumulative conditional variance. Loosely speaking, Freedman’s inequality implies that w.p. $\geq 1 - δ$ ,

{~ ℓ}_{1 : T}^{2} (i) - ℓ_{1 : T}^{2} (i) \leq ~ O ⎛ ⎜ ⎝ \sqrt{T \sum t = 1 Var (Z_{t, i} | F_{t - 1})} ⎞ ⎟ ⎠ .

Importantly, the sum of conditional variances can be related to the regret. Indeed let $p^{*}$ be the best distribution in hindsight, i.e., $p^{*} = arg min \sum_{t = 1}^{T} f_{t} (p)$ , and define

n^{2} {Regret}_{T} (i) = T \sum t = 1 \frac{ℓ_{t}^{2} (i)}{{~ p}_{t} (i)} - T \sum t = 1 \frac{ℓ_{t}^{2} (i)}{p^{*} (i)}

Then the following can be shown,

T \sum t = 1 Var (Z_{t, i} | F_{t - 1}) = ~ O (n^{2} L \cdot {Regret}_{T} (i) + \frac{ℓ_{1 : T}^{2} (i)}{p^{*} (i)}) .

To simplify the proof sketch, ignore the second term. Plugging this back into Freedman’s inequality we get,

{~ ℓ}_{1 : T}^{2} (i) - ℓ_{1 : T}^{2} (i) \leq ~ O (\sqrt{n^{2} L \cdot {Regret}_{T} (i)}) .

(9)

Final bound. Combining the above with the definition of $(A)$ one can to show that w.p. $\geq 1 - δ$ ,

(A) \leq ~ O (n \sqrt{L T} n \sum i = 1 {(n^{2} L \cdot {Regret}_{T} (i))}_{T}^{\frac{1}{4}}) .

Since $(A)$ is bounded by $poly (n, T)$ , we can take a small enough $δ = 1 / poly (n, T)$ such that,

	$E [(A)]$	$\leq ~ O (n^{3 / 2} L^{3 / 4} T^{1 / 2} \cdot E [n \sum i = 1 {({Regret}_{T} (i))}_{T}^{1 / 4}])$
		$\leq ~ O (n^{3 / 2} L^{3 / 4} T^{1 / 2} \cdot n \sum i = 1 {(E [{Regret}_{T} (i)])}^{1 / 4})$
		$\leq ~ O (n^{9 / 4} L^{3 / 4} T^{1 / 2} \cdot {(E [{Regret}_{T}])}^{1 / 4})$

where the second line uses Jensen’s inequality with respect to the concave function $h (u) = u^{1 / 4}$ , and the last line uses $\sum_{i = 1}^{n} {Regret}_{T} (i) = {Regret}_{T}$ together with the fact that $\sum_{i = 1}^{n} x_{i}^{1 / 4} \leq n^{3 / 4} {(\sum_{i = 1}^{n} x_{i})}_{i}^{1 / 4}$ , which is also a consequence of Jensen’s inequality since $\frac{1}{n} \sum_{i = 1}^{n} x_{i}^{1 / 4} \leq {(\frac{1}{n} \sum_{i = 1}^{n})}^{1 / 4}$ . Plugging the above bound back into Eq. (4) we are able to establish the proof. The full proof is deferred to Appendix E. Note that in the full proof we do not explicitly relate the conditional variances to the regret, but this is rather more implicit in the analysis. ∎

5 Experiments

5.1 Image Classification

Training a binary classifier with imbalanced data is a challenging task in machine learning. Practices for dealing with imbalance include optimizing class weight hyperparameters, hard negative mining

(Shrivastava et al., 2016) and synthetic minority oversampling (Chawla et al., 2002). Without accounting for imbalance, the minority samples are often misclassified in early stages of the iterative training procedures, resulting in high loss and high gradient norms associated with these points. Importance sampling schemes for reducing the variance of the gradient norms will sample these instances more often at the early phases, offering a way of tackling imbalance.

For verifying this intuition, we perform the image classification experiment of Bouchard et al. (2015)

. We train one-vs-all logistic regression Pascal VOC 2007 dataset

(Everingham et al., 2010)

with image features extracted from the last layer of the VGG16

(Simonyan and Zisserman, 2015)

pretrained on Imagenet. We measure the average precision by reporting its mean over the 20 classes of the test data. The optimization is performed with AdaGrad

(Duchi et al., 2011), where the learning rate is initialized to 0.1. The losses received by the bandit methods are the norms of the logistic loss gradient. We compare our method, Variance Reducer Bandit (VRB), to:

uniform sampling for SGD,
Adaptive Weighted SGD (AW) (Bouchard et al., 2015) — variance reduction by sampling from a chosen distribution whose parameters are optimized alternatingly with the model parameters,
MABS (Salehi et al., 2017) — bandit algorithm for variance reduction that relies on EXP3 through employing modifies losses.

Figure 3: Mean Average Precision scores achieved on the test part of VOC 2007.

The hyperparameters of the methods are chosen based on cross-validation on the validation portion of the dataset. The results can be seen in Figure 4, where the shaded areas represent confidence $95 %$ intervals over 10 runs. The best performing method is AW, but its disadvantage compared to the bandit algorithms is that it requires choosing a family of sampling distributions, which usually incorporates prior knowledge, and calculating the derivative of the log-density. VRB and AW both outperform uniform subsampling with respect to the training time. VRB performs similarly to AW at convergence, and speeds up training 10 times compared to uniform sampling, by attaining a certain score level 10 times faster. We have also experimented with the variance reduction method of Namkoong et al. (2017), but it did not outperform uniform sampling significantly. Since cross-validation is costly, in Figure 4 we show the effect of the hyperparameters of our method. More specifically, we compare the performance of VRB with misspecified regularizer $L = 1$ to the best $L = 10^{8}$ chosen by cross-validation, and we compensate by using higher mixing coefficient $θ = 0.4$ . The fact that only the early-stage performance is affected is a sign of method’s robustness against regularizer misspecification.

5.2 $k$ -Means

In this experiment, we show that in some applications it is beneficial to work with per-sample upper bound estimates $L_{i}$ instead of a single global bound. As an illustrative example, we choose mini-batch $k$ -Means clustering (Sculley, 2010). This is a slight deviation from the presented theory, since we sample multiple points for the batch and update the sampler only once, upon observing the loss for the batch.

In the case of $k$ -Means, the parameters consist of the coordinates of the $k$ centers $Q = {q_{1}, q_{2}, \dots, q_{k}}$ . As the cost function for a point $x_{i} \in {x_{1}, x_{2}, \dots, x_{n}}$ is the squared Euclidean distance to the closest center, the loss received by VRB is the norm of the gradient ${min}_{q \in Q} 2 \cdot | | x_{i} - q | |_{2}$ . This lends itself to a natural estimation of $L_{i}$ : choose a point $u$ randomly from the dataset and define $L_{i} = 4 \cdot | | x_{i} - u | |_{2}^{2}$ . For this experiment, we set $θ = 0.5$ .

We solve mini-batch $k$ -Means for $k = 100$ and batch size $b = 100$ with uniform sampling and VRB. The initial centers are chosen with $k$ -Means++ (Arthur and Vassilvitskii, 2007) from a random subsample of 1000 points from the training data and they are shared between the methods. We generate 10 different sets of initial centers and run both algorithms 10 times on each set of centers, with different random seeds for the samplers. We train the algorithm on $80 %$ of the data, and measure the cost of the $20 %$ test portion for the following datasets:

CSN (Faulkner et al., 2011) — cellphone accelerometer with 80,000 observations and 17 features,
KDD (KDD Cup 2004, ) — data set used for Protein Homology Prediction KDD competition containing 145,751 observations with 74 features,
MNIST (LeCun et al., 1998) — 70,000 low resolution images of handwritten characters transformed using PCA with whitening and retaining 10 dimensions.

Figure 5: The evolution of the loss of $k$ -Means on the test set. The shaded areas represent $95 %$ confidence intervals over 100 runs.

The evolution of the cost function on the test set with respect to the elapsed training time is shown in Figure 5. The chosen datasets illustrate three observed behaviors of our algorithm. In the case of CSN, our method significantly outperforms uniform subsampling. In the case of KDD, the advantage of our method can be seen in the reduced variance of the cost over multiple runs, whereas on MNIST we observe no advantage. This behavior is highly dependent on intrinsic dataset characteristics: for MNIST, we note that the entropy of the best-in-hindsight sampling distribution is close the entropy of the uniform distribution. We have also compared VRB with the bandit algorithms mentioned in the previous section. Since mini-batch $k$

-Means converges in 1-2 epochs, these methods with uniform initialization do not outperform uniform subsampling significantly. Thus, for this setting, careful initialization is necessary, which is naturally supported by our method.

6 Conclusion and Future Work

We presented a novel importance sampling technique for variance reduction in an online learning formulation. First, we motivated why regret is a sensible measure of performance in this setting. Despite the bandit feedback and the unbounded costs, we provided an expected regret guarantee of $~ O (n^{1 / 3} T^{2 / 3})$ , where we reference is the best fixed sampling distribution in hindsight. We confirmed the theoretical findings with empirical validation.

Among the many possible future directions stands the question of the tightness of the expected regret bound of the algorithm. Another naturally arising idea is theoretical analysis of the method when employed in conjunction with advanced stochastic solvers such as SVRG and SAGA.

Acknowledgement

The authors would like to thank Hasheminezhad Seyedrouzbeh for useful discussions during the course of this work. This research was supported by SNSF grant $407540_167212$ through the NRP 75 Big Data program. K.Y.L. is supported by the ETH Zurich Postdoctoral Fellowship and Marie Curie Actions for People COFUND program.

References

Abernethy et al. (2008) J. Abernethy, E. Hazan, and A. Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In COLT, pages 263–274, 2008.
Alain et al. (2015) G. Alain, A. Lamb, C. Sankar, A. Courville, and Y. Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481, 2015.
Allen-Zhu et al. (2016) Z. Allen-Zhu, Z. Qu, P. Richtárik, and Y. Yuan. Even faster accelerated coordinate descent using non-uniform sampling. In International Conference on Machine Learning, pages 1110–1119, 2016.
Arthur and Vassilvitskii (2007) D. Arthur and S. Vassilvitskii. k-means++: The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027–1035. Society for Industrial and Applied Mathematics, 2007.
Auer et al. (2002) P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48–77, 2002.
Bottou and Bengio (1995) L. Bottou and Y. Bengio. Convergence properties of the k-means algorithms. In Advances in neural information processing systems, pages 585–592, 1995.
Bouchard et al. (2015) G. Bouchard, T. Trouillon, J. Perez, and A. Gaidon. Online learning to sample. arXiv preprint arXiv:1506.09016, 2015.
Cesa-Bianchi et al. (2004) N. Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050–2057, 2004.
Chawla et al. (2002) N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research
, 16:321–357, 2002.
Csiba and Richtárik (2016) D. Csiba and P. Richtárik. Importance sampling for minibatches. arXiv preprint arXiv:1602.02283, 2016.
Defazio et al. (2014) A. Defazio, F. Bach, and S. Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Information Processing Systems, pages 1646–1654, 2014.
Duchi et al. (2011) J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011.
Everingham et al. (2010) M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision
, 88(2):303–338, 2010.
Faulkner et al. (2011) M. Faulkner, M. Olson, R. Chandy, J. Krause, K. M. Chandy, and A. Krause. The next big one: Detecting earthquakes and other rare events from community-based sensors. In Information Processing in Sensor Networks (IPSN), 2011 10th International Conference on, pages 13–24. IEEE, 2011.
Freedman (1975) D. A. Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100–118, 1975.
Hazan (2011) E. Hazan. A survey: The convex optimization approach to regret minimization. Optimization for machine learning, pages 287–302, 2011.
Johnson and Zhang (2013) R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pages 315–323, 2013.
Kakade and Tewari (2009) S. M. Kakade and A. Tewari. On the generalization ability of online strongly convex programming algorithms. In Advances in Neural Information Processing Systems, pages 801–808, 2009.
Kalai and Vempala (2005) A. Kalai and S. Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 71(3):291–307, 2005.
(20) KDD Cup 2004. KDD Cup 2004. Protein Homology Dataset. http://osmot.cs.cornell.edu/kddcup/, 2004. Accessed: 10.11.2016.
LeCun et al. (1998) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
McMahan and Streeter (2010) H. B. McMahan and M. Streeter. Adaptive bound optimization for online convex optimization. COLT 2010, page 244, 2010.
Namkoong et al. (2017) H. Namkoong, A. Sinha, S. Yadlowsky, and J. C. Duchi. Adaptive sampling probabilities for non-smooth optimization. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2574–2583, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.
Necoara et al. (2011) I. Necoara, Y. Nesterov, and F. Glineur. A random coordinate descent method on large optimization problems with linear constraints. 2011.
Needell et al. (2014) D. Needell, R. Ward, and N. Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Advances in Neural Information Processing Systems, pages 1017–1025, 2014.
Nesterov (2012) Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341–362, 2012.
Perekrestenko et al. (2017) D. Perekrestenko, V. Cevher, and M. Jaggi. Faster coordinate descent via adaptive importance sampling. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54. PMLR, 2017.
Salehi et al. (2017) F. Salehi, L. E. Celis, and P. Thiran. Stochastic Optimization with Bandit Sampling. ArXiv e-prints, Aug. 2017.
Salehi et al. (2017) F. Salehi, P. Thiran, and L. E. Celis. Stochastic dual coordinate descent with bandit sampling. arXiv preprint arXiv:1712.03010, 2017.
Sculley (2010) D. Sculley. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, pages 1177–1178. ACM, 2010.
Shalev-Shwartz et al. (2012) S. Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107–194, 2012.
Shrivastava et al. (2016) A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pages 761–769, 2016.
Simonyan and Zisserman (2015) K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Stich et al. (2017) S. U. Stich, A. Raj, and M. Jaggi. Safe adaptive importance sampling. In Advances in Neural Information Processing Systems 30, pages 4384–4394. Curran Associates, Inc., 2017.
Zhao and Zhang (2015) P. Zhao and T. Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1–9, 2015.

Appendix A Cumulative Variance of the Gradients and Quality of Optimization

The relationship between cumulative second moment of the gradients and quality of optimization has been demonstrated in several works. Since the difference between the second moment and the variance is independent of the sampling distribution $p_{t}$ , the guarantees of our method also translate to guarantees with respect to the cumulative second moments of the gradient estimates. Here we provide two concrete references.

For the following, assume that we would like to minimize a convex objective,

min w \in W F (w) := E_{z \sim D} [f (w; z)]

and we assume that we are able to draw i.i.d. samples from the unknown distribution $D$ . Thus, given a point $w \in W$ we are able to design an unbiased estimate for $\nabla F (w)$ by sampling $z \sim D$ and taking $g := \nabla f (w; z)$ (clearly, $E [g | w] = \nabla F (w)$ ). Now assume a gradient-based update rule, i.e.,

w_{t + 1} = Π_{W} (w_{t} - η_{t} g_{t}), where E [g_{t} | w_{t}] = \nabla F (w_{t})

(10)

and $Π_{W} (u) := {a r g m i n}_{w \in W} ∥ u - w ∥$ . Next we show that for two very popular gradient based-methods — AdaGrad and SGD for strongly-convex functions, the performance is directly related to the cumulative second moment of the gradient estimates, $\sum_{t = 1}^{T} E ∥ g_{t} ∥^{2}$ . The latter is exactly the objective of our online variance reduction method.

The AdaGrad algorithm employs the same rule as in Eq. (10) using $η_{t} = D / \sqrt{2 \sum_{τ = 1}^{t} ∥ g_{t} ∥^{2}}$ . The next theorem substantiates its guarantees.

Theorem 9 (Duchi et al. (2011)).

Assume that the diameter of $W$ is bounded by $D$ . Then:

E [F (\frac{1}{T} T \sum t = 1 w_{t})] - min w \in W F (w) \leq \frac{2 D}{T} \sqrt{T \sum t = 1 E ∥ g_{t} ∥^{2}}

The SGD algorithm for $μ$ -strongly-convex objectives employs the same rule as in Eq. (10) using $η_{t} = \frac{2}{μ t}$ . The next theorem substantiates its guarantees.

Theorem 10 (Salehi et al. (2017)).

Assume that $F$ is $μ$ -strongly convex, then:

Appendix B Proof of Lemma 1

Proof.

Denote $ℓ_{1 : t}^{2} (i) = \sum_{τ = 1}^{t} ℓ_{τ}^{2} (i)$ . Next, we bound the cumulative loss per point $i \in [n]$ ,

$ℓ_{1 : T}^{2} (i) = T \sum t = 1 ℓ_{t}^{2} (i)$	$= T \sum t = 1 {(ℓ_{} (i) + (ℓ_{t} (i) - ℓ_{} (i)))}_{*}^{2}$
	$\leq T \cdot ℓ_{}^{2} (i) + 2 ℓ_{} (i) T \sum t = 1 \| ℓ_{t} (i) - ℓ_{} (i) \| + T \sum t = 1 (ℓ_{t} (i) - ℓ_{} (i))^{2}$
	$\leq T \cdot ℓ_{}^{2} (i) + 2 ℓ_{} (i) \sqrt{T \cdot V_{T} (i)} + V_{T} (i)$
	$= T {⎛ ⎝ ℓ_{} (i) + \sqrt{\frac{V_{T} (i)}{T}} ⎞ ⎠}_{}^{2}$	(11)

where the second line uses $ℓ_{*</}$

$ℓ_{1 : T}^{2} (i) = T \sum t = 1 ℓ_{t}^{2} (i)$	$= T \sum t = 1 {(ℓ_{} (i) + (ℓ_{t} (i) - ℓ_{} (i)))}_{*}^{2}$
	$\leq T \cdot ℓ_{}^{2} (i) + 2 ℓ_{} (i) T \sum t = 1 \| ℓ_{t} (i) - ℓ_{} (i) \| + T \sum t = 1 (ℓ_{t} (i) - ℓ_{} (i))^{2}$
	$\leq T \cdot ℓ_{}^{2} (i) + 2 ℓ_{} (i) \sqrt{T \cdot V_{T} (i)} + V_{T} (i)$
	$= T {⎛ ⎝ ℓ_{} (i) + \sqrt{\frac{V_{T} (i)}{T}} ⎞ ⎠}_{}^{2}$	(11)

Online Variance Reduction for Stochastic Optimization

Authors

Related Research

Online Variance Reduction with Mixtures

Variance Reduction for Sequential Sampling in Stochastic Programming

Stochastic Optimization with Bandit Sampling

Query Complexity of Least Absolute Deviation Regression via Robust Uniform Convergence

Adaptive Importance Sampling for Finite-Sum Optimization and Sampling with Decreasing Step-Sizes

Stochastic Primal Dual Coordinate Method with Non-Uniform Sampling Based on Optimality Violations

Learning from eXtreme Bandit Feedback

Code Repositories