Online Convex Optimization with Unbounded Memory

1 Introduction

Online learning models sequential decision-making problems, where a learner must choose a sequence of decisions while interacting with the environment. We refer the reader to cesabianchi2006prediction; shalevshwartz2012online for more background. One of the most popular frameworks in online learning is online convex optimization (OCO) (see, for example, hazan2019oco).

The OCO framework is as follows. In each round, the learner chooses a decision from some convex set and an adversary chooses a convex loss function, and then the learner suffers the loss associated with their current decision. The performance of an algorithm is measured by regret: the difference between the algorithm’s total loss and that of the best fixed decision. This framework has numerous applications, such as prediction from expert advice (littlestone1989weighted; littlestone1994weighted), portfolio selection (cover1991universal), routing (awerbuch2008online), etc. There has been a lot of work in developing a variety of algorithms and proving optimal regret bounds, and we refer the reader to hazan2019oco for a survey. In many applications of the OCO framework the loss of the learner depends not only on the current decisions but on the entire history of decisions until that point. For example, consider the problem of online linear control (agarwal2019online), where in each round the learner chooses a “control policy” (i.e., decision), suffers a loss that is a function of the action taken by the chosen policy and the current state of the system, and the system’s state evolves according to linear dynamics. Note that the current state depends on the entire history of actions and, therefore, the current loss depends not only on the current decision but the entire history of decisions. The OCO framework fails to capture such long-term dependence of the current loss on the past decisions, as do existing generalizations of OCO that allow the loss to depend on a constant number of past decisions (anava2015online). Although a series of approximation arguments can be used to apply the finite memory generalization of OCO to the online linear control problem, there is no OCO framework that captures the complete long-term dependence of current losses on past decisions.

Contributions In this paper we introduce a generalization of the OCO framework, “Online Convex Optimization with Unbounded Memory”, that allows the loss in the current round to depend on the entire history of decisions until that point. We introduce the notion of $p$ -effective memory capacity, $H_{p}$ , that quantifies the maximum influence of past decisions on present losses. We prove a $O (\sqrt{H_{1} T})$ policy regret bound and a stronger $O (\sqrt{H_{p} T})$ policy regret bound under mild additional assumptions about the history (Section 3). Both bounds use the notion of policy regret with respect to constant action sequences, as defined in dekel2012online. Our bounds are optimal in terms of their dependence on the time horizon $T$ , and we show that our results match existing ones for the special case of OCO with finite memory (Section 4.1). Finally, we show the broad applicability of our framework to derive regret bounds, and to simplify existing regret bound derivations, for online learning problems including an online variant of performative prediction and online linear control (Section 4.3 and 4.4).

Related work The most closely related work to ours is the OCO with finite memory framework (anava2015online). They consider a generalization of the OCO framework that allows the current loss to depend on a constant number of past decisions. There have been a number of follow-up works that extend the framework in a variety of other ways, such as non-stationarity (zhao2020nonstationary), incorporating switching costs (shi2020online), etc. However, none of the existing works go beyond a constant memory length. In contrast, our framework allows the current loss to depend on an unbounded number of past decisions.

Reinforcement learning (sutton2018reinforcement) is another popular framework for sequential decision-making that considers very general state-action models of feedback and dynamics. In reinforcement learning one typically measures regret with respect to the best state-action policy from some policy class, rather than the best fixed decision as in online learning and OCO. In the special case of linear control, policies can be reformulated as decisions while preserving convexity; we discuss this application in Section 4. Considering the general framework is an active area of research. For example bhatia2020online provide only non-constructive upper bounds on regret.

We defer discussion of related work for specific applications to Section 4.

2 Framework

Let $T$ denote the time horizon. Let $\calX$ be a closed and convex subset of a Hilbert space, $\calH$ be a Banach space, $A:\calH→\calH$ and $B:\calX→\calH$ be linear operators, and $ft:\calH→[0,1]$ be loss functions chosen by an oblivious adversary. The game between the learner and the adversary proceeds as follows. Let the initial history be $h_{0} = 0$ . In each round $t \in [T]$ , the learner chooses $xt∈\calX$ , the history $h_{t}$ is updated according to $h_{t} = A h_{t - 1} + B x_{t}$ , and the learner suffers loss $f_{t} (h_{t})$ .

Consider a strategy that chooses the same action in every round, i.e. $x_{t} = x$ for some $x∈\calX$ and for all rounds $t$ . The history after round $t$ for such a strategy is described by $h_{t} = \sum_{k = 0}^{t - 1} A^{k} B x$ , which motivates the following definition.

Given a function $ft:\calH→[0,1]$ , the function $\lx@overaccentset∘ft:\calX→[0,1]$ is defined by $\lx@overaccentset∘ft(x)=ft(∑t−1k=0AkBx)$ .

We define the policy regret of a learner as the difference between its total loss and the total loss of a strategy that plays the best fixed action in every round (dekel2012online). [Policy Regret] The policy regret of an algorithm $\calA$ is defined as

RT(\calA)=T∑t=1ft(ht)−minx∈\calXT∑t=1\lx@overaccentset∘ft(x).

(1)

2.1 Notation

We use $∥⋅∥\calX$ to denote the norm associated with the Hilbert space $\calX$ and $∥⋅∥\calH$ to denote the norm associated with the Banach space $\calH$ . The operator norm for a linear $L$ operator from space $U \to V$ is defined as $∥ L ∥_{U \to V} = {max}_{∥ u ∥_{U} \leq 1} ∥ L u ∥_{V}$ . For convenience, sometimes we simply use $∥ \cdot ∥$ when the meaning is clear from the context. For a finite-dimensional matrix, we use $∥ \cdot ∥_{F}, ∥ \cdot ∥_{2}$ , and $σ_{j} (\cdot)$ to denote its Frobenius norm, operator norm, and $j$

-th singular value respectively. When we say that

\calH

is a sequence space with the

∥ \cdot ∥_{p}

norm for

p \geq 1

, we mean that

∥(y0,y1,…)∥\calH=(∑∞k=0∥yk∥p)1p

2.2 Assumptions

We make the following assumptions on $f_{t}$ and $\lx@overaccentset∘ft$ .

[label=A0]
The functions $f_{t}$ are $L$ -Lipschitz continuous, i.e.,

$|ft(h)−ft(~h)|≤L∥h−~h∥\calH.$
The functions $\lx@overaccentset∘ft$ are differentiable. We use $∇\lx@overaccentset∘ft(x)$ to denote its gradient at $x$ . Furthermore, the gradients are bounded, i.e., for all $x∈\calX$ ,

$∥∇\lx@overaccentset∘ft(x)∥≤L.$
The functions $\lx@overaccentset∘ft$ are convex, i.e., for all $x,y∈\calX$ ,

$\lx@overaccentset∘f(y)≥\lx@overaccentset∘f(x)+⟨∇\lx@overaccentset∘ft(x),y−x⟩.$

Assumptions 1, 2, and 3 are standard in the optimization literature. Given these, we assume the following feedback model for the learner.

[label=A0]
There exists a gradient oracle such that at the end of each round $t$ , the learner receives the gradient $∇\lx@overaccentset∘ft(xt)$ .

Assumption 4 is a standard assumption in the literature on first-order optimization methods. Before presenting our final assumption, we introduce the notion of $p$ -effective memory capacity that quantifies the maximum influence of past decisions on present losses.

[ $p$ -Effective Memory Capacity] Consider an online convex optimization with unbounded memory problem specified by the Hilbert space $\calX$ , Banach space $\calH$ , and linear operators $A:\calH→\calH$ and $B:\calX→\calH$ . For $p \geq 1$ , the $p$ -effective memory capacity of this problem is defined as

Hp(\calX,\calH,A,B)=(∞∑k=0kp∥AkB∥p)1p.

(2)

When the meaning is clear from the context we use simply use $H_{p}$ to denote the $p$ -effective memory capacity. One way to understand this definition is the following. Consider two sequences of decisions whose elements differ by no more than $k$ at time $k$ : $∥ x_{k} - {~ x}_{k} ∥ \leq k$ . Then the histories generated by the the sequences have difference bounded as $∥ h - ~ h ∥ = ∥ \sum_{k} A^{k} B (x_{k} - {~ x}_{k}) ∥ \leq \sum_{k} k ∥ A^{k} B ∥ = H_{1}$ . A similar bound holds with $H_{p}$ instead when $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm. Therefore, the $p$ -effective memory capacity is an upper bound on the difference in histories for two sequences of decisions whose difference grows at most linearly in time.

[label=A0]
The $1$ -effective memory capacity is well-defined, i.e., $H_{1} < \infty$ .

When $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm for $p \geq 1$ , then note that $H_{p} \leq H_{1} < \infty$ is well-defined under assumption 5. In this case we obtain stronger regret bounds that depend on $H_{p}$ instead of $H_{1}$ .

3 Regret Analysis

In this section we present a regret analysis for OCO with unbounded memory. The approach consists of writing the regret as

	$RT(\calA)=T∑t=1ft(ht)−minx∈\calXT∑t=1\lx@overaccentset∘ft(x)$
	$=T∑t=1ft(ht)−\lx@overaccentset∘ft(xt)+T∑t=1\lx@overaccentset∘ft(xt)−minx∈\calXT∑t=1\lx@overaccentset∘ft(x)$
	$≤T∑t=1ft(ht)−\lx@overaccentset∘ft(xt)(a)+maxx∈\calX(T∑t=1⟨∇\lx@overaccentset∘ft(xt),xt−x⟩)(b),$

where the last inequality follows from the convexity of $\lx@overaccentset∘ft$ (assumption 3). The second term (b) can be bounded by using the follow-the-regularized-leader (FTRL) algorithm (Sections 3, 3 and 3), which we present in the context of our framework in Algorithm 1. The main novelty comes from bounding the first term (a) by measuring the difference betwen the histories $h_{t}$ and $\sum_{k = 0}^{t - 1} A^{k} B x_{t}$ in terms of the $p$ -effective memory capacity (Section 3).

Let $R:\calX→\R$ be an $α$ -strongly-convex regularizer. Since we assume $\calX$ is a Hilbert space, such an $α$ -strongly-convex regularizer exists. The FTRL algorithm (DBLP:conf/colt/AbernethyHR08; DBLP:conf/colt/Shalev-ShwartzS06; hazan2019oco) chooses $x_{t}$ as the minimizer of the past losses plus a regularizer. In our framework, this corresponds to

xt∈\argminx∈\calXt−1∑s=1⟨∇\lx@overaccentset∘fs(xs),x⟩+R(x)η.

For notational convenience, let $g0(x)=R(x)η,gt(x)=⟨∇\lx@overaccentset∘ft(xt),x⟩$ for $t \geq 1$ , and $x∗∈\argminx∈\calX∑Tt=1gt(x)$ .

Input : Time horizon

T

, step size

η

α

-strongly-convex regularizer

R:\calX→\R

1 Initialize history

h_{0} = 0

2 for $t = 1, 2, \dots, T$ do

3 Learner chooses

xt∈\argminx∈\calX∑t−1s=1⟨∇\lx@overaccentset∘fs(xs),x⟩=gs(x)+R(x)η=g0(x)

4 Set

h_{t} = A h_{t - 1} + B x_{t}

5 Learner suffers loss

f_{t} (h_{t})

and observes

∇\lx@overaccentset∘ft(xt)=∇gt(xt)

6 end for

Algorithm 1 FTRL

Consider an online convex optimization with unbounded memory problem specified by the decision space $\calX$ , history space $\calH$ , and linear operators $A:\calH→\calH$ and $B:\calX→\calH$ . Let the regularizer $R:\calX→\R$ be $α$ -strongly-convex and satisfy $| R (x) - R (x^{'}) | \leq D$ for all $x,x′∈\calX$ . Algorithm 1 with step-size $η$ satisfies the following regret bound

R_{T} (FTRL) \leq \frac{D}{η} + η \frac{T L^{2}}{α} + η \frac{T L^{2}}{α} H_{1} .

(3)

If we set $η = \sqrt{\frac{D α}{T L^{2} (1 + H_{1})}}$ , then the regret bound satisfies

R_{T} (FTRL) \leq 3 \sqrt{\frac{D L^{2} (1 + H_{1})}{α}} \sqrt{T} .

(4)

When $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm for $p \geq 1$ , then the above bounds hold with $H_{p}$ instead of $H_{1}$ .

We prove this theorem using a sequence of lemmas that formalize the approach mentioned at the start of this section. The proofs of Sections 3, 3 and 3 are standard in the literature on FTRL, but we include them in the supplementary material for completeness (Appendix A).

$∀x∈\calX$ , $\sum_{t = 0}^{T} g_{t} (x) \geq \sum_{t = 0}^{T} g_{t} (x_{t + 1})$ .

If the regularizer $R:\calX→\R$ satisfies $R (x) - R (x^{'}) | \leq D$ for all $x,x′∈\calX$ , then Algorithm 1 with step-size $η$ satisfies

T \sum t = 1 g_{t} (x_{t}) - T \sum t = 1 g_{t} (x^{*}) \leq T \sum t = 1 g_{t} (x_{t}) - g_{t} (x_{t + 1}) + \frac{D}{η} .

If the regularizer $R:\calX→\R$ is $α$ -strongly-convex and satisfies $| R (x) - R (x^{'}) | \leq D$ for all $x,x′∈\calX$ , then Algorithm 1 with step-size $η$ satisfies $∥ x_{t + 1} - x_{t} ∥ \leq η \frac{L}{α}$ and

T \sum t = 1 g_{t} (x_{t}) - T \sum t = 1 g_{t} (x^{*}) \leq \frac{D}{η} + η \frac{T L^{2}}{α} .

The following lemma bounds the difference between $f_{t} (h_{t})$ and $\lx@overaccentset∘ft(xt)$ in terms of the $p$ -effective memory capacity (Section 2.2) using the Lipschitz continuity of $f_{t}$ (assumption 1) and the bound on $∥ x_{t + 1} - x_{t} ∥$ from Section 3.

|ft(ht)−\lx@overaccentset∘ft(xt)|≤ηL2αH1

(5)

for all rounds $t$ . When $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm for $p \geq 1$ , then the above bound holds with $H_{p}$ instead of $H_{1}$ .

Proof.

By our Lipschitz continuity assumption (assumption 1), we have

$∣∣ft(ht)−\lx@overaccentset∘ft(xt)∣∣$	$= ∣ ∣ ∣ ∣ f_{t} (h_{t}) - f_{t} (t - 1 \sum k = 0 A^{k} B x_{t}) ∣ ∣ ∣ ∣$
	$\leq L ∥ ∥ ∥ ∥ h_{t} - t - 1 \sum k = 0 A^{k} B x_{t} ∥ ∥ ∥ ∥$
	$\leq L ∥ ∥ ∥ ∥ t - 1 \sum k = 0 A^{k} B x_{t - k} - t - 1 \sum k = 0 A^{k} B x_{t} ∥ ∥ ∥ ∥$	(6)
		(7)

where Eq. 6 follows by definition of $h_{t}$ . In the general case we can bound the term (a) as

$∥ ∥ ∥ ∥ t - 1 \sum k = 0 A^{k} B (x_{t} - x_{t - k}) ∥ ∥ ∥ ∥$	$\leq t - 1 \sum k = 0 ∥ A^{k} B ∥ ∥ x_{t} - x_{t - k} ∥$
	$\leq t - 1 \sum k = 0 ∥ A^{k} B ∥ k η \frac{L}{α}$	(8)
	$\leq η \frac{L}{α} H_{1},$

where Eq. 8 follows by Section 3. Combining this with Eq. 7 completes the proof for the general case.

When $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm for $p \geq 1$ , then we can bound the term (a) as

$∥ ∥ ∥ ∥ t - 1 \sum k = 0 A^{k} B (x_{t} - x_{t - k}) ∥ ∥ ∥ ∥$	$= {(t - 1 \sum k = 0 ∥ A^{k} (x_{t} - x_{t - k}) ∥^{p})}_{t}^{\frac{}{1} p}$
	$\leq {(t - 1 \sum k = 0 ∥ A^{k} ∥^{p} ∥ x_{t} - x_{t - k} ∥^{p})}_{t}^{\frac{1}{p}}$
	$\leq η \frac{L}{α} {(t - 1 \sum k = 0 k^{p} ∥ A^{k} ∥^{p})}^{\frac{1}{p}}$	(9)
	$\leq η \frac{L}{α} {(t - 1 \sum k = 0 k^{p} ∥ A^{k} B ∥^{p})}^{\frac{1}{p}}$	(10)
	$\leq η \frac{L}{α} H_{p},$

where Eq. 9 follows by Section 3 and Eq. 10 follows because $B (x) = (x, 0, 0, \dots)$ . Combining this with Eq. 7 completes the proof for the case when $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm for $p \geq 1$ . ∎

Proof of Section 3.

As stated at the start of this section, we can write the regret as

	$R_{T} (FTRL)$	$≤T∑t=1ft(ht)−\lx@overaccentset∘ft(xt)(a)+T∑t=1gt(xt)−minx∈\calXT∑t=1gt(x)(b)$
		$\leq η \frac{T L^{2}}{α} H_{1} + \frac{D}{η} + η \frac{T L^{2}}{α},$

where the last inequality follows by bounding term (a) using Section 3 and bounding term (b) using Section 3. When $\calH$ is a sequence space with the $∥ \cdot ∥_{p}$ norm for $p \geq 1$ , we can bound term (a) using Section 3 by $η \frac{T L^{2}}{α} H_{p}$ instead. This completes the proof. ∎

4 Applications

4.1 Special Case: OCO with Finite Memory

We start by showing how OCO with finite memory (anava2015online) is a special case of our framework. ¹¹1The only difference is that the summation in our definition of regret starts at $t = 1$ whereas it starts at $t = m$ in the OCO with finite memory framework. However, this difference only adds $O (m)$ to the regret and $m$ is a constant in the OCO with finite memory framework, so we can ignore it. Formally, let $\calX⊆\Rd$ be closed and convex with $∥⋅∥\calX=∥⋅∥2$ . Let $\calH=\calXm$ with $∥h∥\calH=√∑mj=1∥xj∥2$ , where $h = (x_{0}, \dots, x_{m})$ . Define the linear operators as $A:\calH→\calH$ and $B:\calX→\calH$ as $A (x_{t}, \dots, x_{t - m}) = (0, x_{t}, \dots, x_{t - m + 1})$ and $B (x) = (x, 0, \dots, 0)$ respectively. In words, the history $h_{t} = (x_{t}, \dots, x_{t - m})$ consists of the past $m$ decisions and gets updated as $h_{t + 1} = (x_{t + 1}, x_{t}, \dots, x_{t - m + 1})$ . Note that $\calH$ is a sequence space with the $∥ \cdot ∥_{2}$ norm. Therefore, we can apply Section 3 with $p = 2$ . It remains to bound $H_{2}$ .

Let $x∈\calX$ with $∥x∥\calX=1$ and $h∈\calH$ with $∥h∥\calH=1$ . By definition, we have $∥Bx∥\calH=∥x∥\calX=1$ , for $k > m$ we have $∥ A^{k} h ∥ = ∥ 0 ∥ = 0$ , and for $k \leq m$ we have $∥ A^{k} h ∥ \leq ∥ h ∥ = 1$ . Therefore, $H_{2} \leq \sqrt{\sum_{k = 1}^{m} k^{2}} \leq O (m^{3 / 2})$ . Plugging this into Section 3, we obtain a $O (m^{3 / 4} \sqrt{T})$ regret bound that matches existing results (anava2015online).

4.2 OCO with Infinite Memory

We now show how our framework can model OCO with infinite memory problems that are not modelled by existing works. Formally, let $\calX⊆\Rd$ be closed and convex with $∥⋅∥\calX=∥⋅∥2$ . Let $\calH⊆\calX\N$ consist of $h = (y_{0}, y_{1}, \dots)$ satisfying $\sum_{j = 0}^{\infty} ∥ y_{j} ∥^{2} < \infty$ with $∥h∥\calH=√∑∞j=0∥yj∥2$ . Let $ρ \in (0, 1)$ . Define the linear operators $A:\calH→\calH$ and $B:\calX→\calH$ as $A ((y_{0}, y_{1}, y_{2}, \dots)) = (0, ρ y_{0}, ρ y_{1}, \dots)$ and $B (x) = (x, 0, 0, \dots)$ respectively. Note that $\calH$ is a sequence space with the $∥ \cdot ∥_{2}$ norm. Therefore, we can apply Section 3 with $p = 2$ . It remains to bound $H_{2}$ .

Let $x∈\calX$ with $∥x∥\calX=1$ and $h=(y0,y1,…)∈\calH$ with $∥h∥\calH=1$ . By definition, we have that $∥Bx∥\calH=∥x∥\calX=1$ , $A^{k} h = (0, \dots, 0, ρ^{k} y_{0}, ρ^{k} y_{1}, \dots)$ and $∥ A^{k} h ∥ = ρ^{k}$ . Therefore,

H_{2} \leq \sqrt{\infty \sum k = 1 k^{2} ρ^{2 k}} \leq O ⎛ ⎜ ⎝ \frac{ρ^{4}}{(1 - ρ^{2})^{\frac{3}{2}}} ⎞ ⎟ ⎠ .

A useful extension of the above is when elements of $\calX$ are $d \times d$ matrices and the linear operator $A$ is defined as $A ((y_{0}, y_{1}, \dots)) = (0, Z y_{0}, Z y_{1}, \dots)$ , where $Z∈\Rd×d$ satisfies $∥ Z ∥_{2} < 1$ . An even more general extension of the above is the crux of our simplified regret bounds for online linear control (Section 4.4).

4.3 Online Performative Prediction

4.3.1 Background

In many applications of machine learning the algorithm’s decisions influence the data distribution, for example, online labor markets

(DBLP:conf/kdd/Anagnostopoulos18; horton2010online), predictive policing (lum2016predict), on-street parking DBLP:journals/tits/DowlingRZ20; pierce2018sfpark, and vehicle sharing markets (banerjee2015pricing) to name a few. Motivated by such applications, several works have studied the problem of performative prediction, which models the data distribution as a function of the decision-maker’s decision (DBLP:conf/icml/PerdomoZMH20; DBLP:conf/nips/Mendler-DunnerP20; DBLP:conf/icml/MillerPZ21; DBLP:conf/aistats/BrownHK22; DBLP:conf/aaai/RayRDF22). These works view the problem as a stochastic optimization problem; we refer the reader to these citations for more details. As a natural extension to existing works, we introduce an online learning variant of performative prediction with geometric decay (DBLP:conf/aaai/RayRDF22) that differs from the original formulation in a few key ways.

Let $\calX⊆\Rd$ denote the decision set with $∥x∥2≤R\calX$ . Let $p_{0}$ denote the initial data distribution over the instance space $\calZ$ . In each round $t \in [T]$ , the learner chooses a decision $xt∈\calX$ and an oblivious adversary chooses a loss function $lt:\calX×\calZ→[0,1]$ , and then the learner suffers the loss $Lt(xt)=\Ez∼pt[lt(xt,z)]$ , where $p_{t} = p_{t} (x_{1}, \dots, x_{t})$ is the data distribution in round $t$ . We also use the shorthand $p_{t} (x) = p_{t} (x, \dots, x)$ .

The goal in our online learning setting is to minimize the difference between the algorithm’s total loss and the total loss of the best fixed decision,

T∑t=1\Ez∼pt[lt(xt,z)]−minx∈\calXT∑t=1\Ez∼pt(x)[lt(x,z)],

as a natural generalization of performative optimality (DBLP:conf/icml/PerdomoZMH20) for our online learning formulation.

We make the following assumptions about the distributions and loss functions. The data distribution satisfies $p_{1} = p_{0}$ , and for all $t \geq 1$ , $pt+1=ρpt+(1−ρ)\calD(xt)$ , where $ρ \in (0, 1)$ and $\calD(xt)$ is a distribution over $\calZ$ that depends on the decision $x_{t}$ . The distributions $\calD(x)$ are assumed to belong to the following location-scale family: $z∼\calD(x)$ iff $z \sim ξ + A x$ , where $A∈\Rd×d$ satisfies $∥ A ∥_{2} < \infty$ and $ξ$

is a random variable with mean

μ

and covariance

Σ

. The loss functions

l_{t}

are assumed to be convex, differentiable, and

\wtL

-Lipschitz continuous.

Our problem formulation differs from existing work in the following ways. First, we adopt an online learning perspective, whereas DBLP:conf/aaai/RayRDF22 adopt a stochastic optimization perspective. Therefore, we assume that the loss functions $l_{t}$ are adversarially chosen, whereas DBLP:conf/aaai/RayRDF22 assume $l_{t} = l$ are fixed. Second, our gradient oracle (assumption 4) assumes that the dynamics ( $\calD$ and $ρ$ ) are known, whereas DBLP:conf/aaai/RayRDF22estimate the gradient using samples from this distribution. We believe that an appropriate extension of our framework that can deal with unknown linear operators $A$ and $B$ can be applied to this more difficult setting, and we leave this as future work.

Before formulating this problem in our OCO with unbounded memory framework, we state the definition of $1$ -Wasserstein distance that we use in our regret analysis. Informally, the $1$

-Wasserstein distance is a measure of the distance between two probability measures.

[1-Wasserstein Distance] Let $(\calZ,d)$ be a metric space. Let $P(\calZ)$ denote the set of Radon probability measures $ν$ on $\calZ$

with finite first moment. That is, there exists

z′∈\calZ

such that

\Ez∼ν[d(z,z′)]<∞

. The

1

-Wasserstein distance between two probability measures

ν,ν′∈P(\calZ)

is defined as

W1(ν,ν′)=sup{\Ez∼ν[f(z)]−\Ez∼ν′[f(z)]},

where the supremum is taken over all $1$ -Lipschitz continuous functions $f:\calZ→\R$ .

4.3.2 Formulation as OCO with Unbounded Memory

Let $\calX⊆\Rd$ be closed and convex with $∥⋅∥\calX=∥⋅∥2$ . Let $\calH⊆\calX\N$ consist of sequences $h = (x_{0}, ρ x_{1}, ρ^{2} x_{2}, \dots)$ , where $ρ \in (0, 1)$ and $xk∈\calX$ satisfy $\sum_{k = 0}^{\infty} ∥ x_{k} ∥^{2} < \infty$ , with $∥h∥\calH=√∑∞k=0ρ2k∥xk∥2$ .

Define the linear operators $A:\calH→\calH$ and $B:\calX→\calH$ as $A ((y_{0}, y_{1}, y_{2}, \dots)) = (0, ρ y_{0}, ρ y_{1}, \dots)$ and $B (x) = (x, 0, 0, \dots)$ respectively. Note that $\calH$ is a sequence space with the $∥ \cdot ∥_{2}$ norm.

By definition, given a sequence of decisions $(x_{1}, \dots, x_{t})$ , the history at the end of round $t$ is given by $h_{t} = (x_{t}, ρ x_{t - 1}, ρ^{2} x_{t - 2}, \dots, ρ^{t} x_{1}, 0, \dots)$ . Define the functions $ft:\calH→[0,1]$ by $ft(ht)=\Ez∼pt(x1,…,xt)[lt(xt,z)]$ , where $p_{t} (x_{1}, \dots, x_{t})$ is the distribution that satisfies $z \sim p_{t}$ iff

z \sim t - 1 \sum k = 1 (1 - ρ) ρ^{k - 1} (ξ + A x_{t - k}) + ρ^{t} p_{0} .

(11)

Note that the distribution $p_{t}$ is a function of the history $h_{t}$ . The above follows from the recursive definition of $p_{t}$ and parametric assumption about $\calD(x)$ .

With the above formulation and definition of $f_{t}$ , the original goal of minimizing the difference between the algorithm’s total loss and the total loss of the best fixed decision is equivalent to minimizing the regret,

T∑t=1ft(ht)−minx∈\calXT∑t=1\lx@overaccentset∘ft(x).

4.3.3 Regret Analysis

In order to apply our regret bounds, we must bound $H_{2}$ , the Lipschitz continuity constant (assumption 1), and the norm of the gradients (assumption 2).

We have $H_{2} \leq O (\frac{ρ^{4}}{(1 - ρ^{2})^{\frac{3}{2}}}) < \infty$ .

Proof.

Note that $∥ B ∥ = 1$ and $∥ A^{k} ∥ \leq ρ^{k}$ . Therefore,

H2= ⎷∞∑k=0k2∥AkB∥2≤ ⎷∞∑k=0k2ρ2k≤O⎛⎜⎝ρ4(1−ρ2)32⎞⎟⎠.\qed

We have the following bound on the Lipschitz continuity constant: $L≤\wtL1−ρρ∥A∥2$ .

Proof.

Let $(x_{0}, \dots, x_{t})$ and $(\wtx1,…,\wtxt)$ be two sequences of decisions, where $xk,\wtxk∈\calX$ . Let $h_{t}$ and $\wtht$ be the corresponding histories, and $p_{t}$ and $\wtpt$ be the corresponding distributions at the end of round $t$ . We have

	$∣∣ft(ht)−ft(\wtht)∣∣$	$=∣∣\Ez∼pt[lt(xt,z)]−\Ez∼\wtpt[lt(\wtxt,z)]∣∣$
		$=\|\Ez∼pt[lt(xt,z)]−\Ez∼pt[lt(\wtxt,z)]$
		$+\Ez∼pt[lt(\wtxt,z)]−\Ez∼\wtpt[lt(\wtxt,z)]\|$
		$≤\wtL∥xt−\wtxt∥2+\wtLW1(pt,\wtpt),$

where the last inequality follows from the assumptions about the functions $l_{t}$ and the definition of the Wasserstein distance $W_{1}$ . By definition of $p_{t}$ Eq. 11, we have that

	$W1(pt,\wtpt)$	$≤t−1∑k=11−ρρρk∥A∥2∥xt−k−\wtxt−k∥2$
		$≤1−ρρ∥A∥2∥ht−\wtht∥2,$

where the last inequality follows from the definition of $∥⋅∥\calH$ . Therefore, $L≤\wtL1−ρρ∥A∥2$ . ∎

We have the following bound on the norm of the gradient: $∥∥∇\lx@overaccentset∘ft(x)∥∥≤\wtL(1+∥A∥2)$ .

Proof.

We have (DBLP:conf/aaai/RayRDF22)

∇\lx@overaccentset∘ft(x)=\Ez∼pt[∇xlt(x,z)+(1−ρt)A⊤∇zlt(x,z)].

The results follows because the functions $l_{t}$ are assumed to be $\wtL$ -Lipschitz continuous in $(x, z)$ . ∎

For the online performative prediction problem, we have

RT(FTRL)≤O⎛⎜⎝R\calX\wtLL′1−ρρ ⎷1+ρ4(1−ρ2)32√T⎞⎟⎠,

where $L^{'} = max {1 + ∥ A ∥_{2}, \frac{1 - ρ}{ρ} ∥ A ∥_{2}}$ .

Proof.

Choosing the $R(⋅)=12∥⋅∥2\calX$ as the $1$ -strongly convex regularizer, we have that for all $x,x′∈\calX$ ,

|R(x)−R(x′)|≤R2\calX=D.

Recall the regret bound from Section 3. Using the above bound on $D$ and using Sections 4.3.3, 4.3.3 and 4.3.3 to bound the other constants, we obtain the result. ∎

4.4 Online Linear Control

4.4.1 Background

Now we use our framework to simplify regret bound derivations for online linear control (agarwal2019online). Existing works model it (and its extensions, such as, DBLP:conf/nips/AgarwalHS19; DBLP:conf/alt/HazanKS20; DBLP:conf/aaai/LiD021 to name a few) as an OCO with finite memory problem. This necessitates a lot of truncation and error analysis that constitutes the bulk of existing proofs. On the other hand, this problem fits very naturally into our framework because it is inherently an OCO with unbounded memory problem. We refer the reader to existing literature (agarwal2019online) for more details and background on online linear control. Here, we introduce the basic mathematical setup of the problem.

Let $\calS⊆\Rds$ denote the state space, $\calU⊆\Rdu$ denote the control space, and $s_{t}$ and $u_{t}$ denote the state and control at time $t$ with $s_{0}$ being the inital state. The system evolves according to the linear dynamics $s_{t + 1} = F s_{t} + G u_{t} + w_{t}$ , where $F∈\Rds×ds,G∈\Rds×du$ are matrices satisfying $∥ F ∥_{2}, ∥ G ∥_{2} \leq κ$ and $wt∈\Rds$ is an adversarially chosen “disturbance” with $∥ w ∥_{\infty} \leq W$ . We assume, without loss of generality and for notational convenience, that $d_{s} = d_{u} = d$ and also define $w_{- 1} = s_{0}$ .

For $t = 0, \dots, T - 1$ , let $ct:\calS×\calU→[0,1]$ be convex loss functions chosen by an oblivous adversary. The functions $c_{t}$ are assumed to satisfy the following Lipschitz condition: if $∥s∥2,∥u∥2≤D\calX$ , then $∥∇sct(s,u)∥,∥∇uct(s,u)∥≤\wtLD\calX$ . The goal in online linear control is to choose a sequence of controls $u_{t}$ to minimize the following regret

R_{T} (Π) = T - 1 \sum t = 0 c_{t} (s_{t}, u_{t}) - min π^{*} \in Π T - 1 \sum t = 0 c_{t} (s_{t}^{π^{*}}, u_{t}^{π^{*}}),

(12)

where $s_{t}$ evolves according to linear dynamics as stated above, $Π$ denotes an arbitrary policy class, and $s_{t}^{π^{*}}, c_{t}^{π^{*}}$ denote the state and control at time $t$ when the controls are chosen according to the policy $π^{*}$ .

Like previous work (agarwal2019online) we focus on the policy class of $(κ, ρ)$ -strongly stable linear controllers, $\calK$ , where $K∈\calK$ satisfies $F - G K = H L H^{- 1}$ with $∥ K ∥_{2}, ∥ H ∥_{2}, ∥ H^{- 1} ∥_{2} \leq κ$ and $∥ L ∥_{2} = ρ < 1$ . Given such a controller, the controls are chosen as linear functions of the current state, i.e., $u_{t} = - K s_{t}$ . The motivation for considering this policy class comes from the fact that for an infinite horizon problem with quadratic costs, the optimal controller is a fixed linear controller.

Unfortunately, parameterizing $u_{t}$ directly with a linear controller as $u_{t} = - K s_{t<}$

Online Convex Optimization with Unbounded Memory

Online non-convex optimization with imperfect feedback

Online Convex Optimization Perspective for Learning from Dynamically Revealed Preferences

Online Multiobjective Minimax Optimization and Applications

Online learning with dynamics: A minimax perspective

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Optimistic and Adaptive Lagrangian Hedging

Zeroth-order non-convex learning via hierarchical dual averaging

1 Introduction

2 Framework

2.1 Notation

2.2 Assumptions

3 Regret Analysis

Proof.

Proof of Section 3.

4 Applications

4.1 Special Case: OCO with Finite Memory

4.2 OCO with Infinite Memory

4.3 Online Performative Prediction

4.3.1 Background

4.3.2 Formulation as OCO with Unbounded Memory

4.3.3 Regret Analysis

Proof.

Proof.

Proof.

Proof.

4.4 Online Linear Control

4.4.1 Background

Online Convex Optimization with Unbounded Memory

Related Research

Online non-convex optimization with imperfect feedback

Online Convex Optimization Perspective for Learning from Dynamically Revealed Preferences

Online Multiobjective Minimax Optimization and Applications

Online learning with dynamics: A minimax perspective

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Optimistic and Adaptive Lagrangian Hedging

Zeroth-order non-convex learning via hierarchical dual averaging

1 Introduction

2 Framework

2.1 Notation

2.2 Assumptions

3 Regret Analysis

Proof.

Proof of Section 3.

4 Applications

4.1 Special Case: OCO with Finite Memory

4.2 OCO with Infinite Memory

4.3 Online Performative Prediction

4.3.1 Background

4.3.2 Formulation as OCO with Unbounded Memory

4.3.3 Regret Analysis

Proof.

Proof.

Proof.

Proof.

4.4 Online Linear Control

4.4.1 Background