A relaxed technical assumption for posterior sampling-based reinforcement learning for control of unknown linear systems

I Introduction

In a recent paper, Ouyang et al. [1] presented a Thompson sampling (also called posterior sampling) algorithm to control a linear quadratic (LQ) system with unknown parameters. Their algorithm is called Thompson sampling with dynamic episodes (TSDE).¹¹1In [1], the algorithm is called posterior sampling reinforcement learning (PSRL-LQ). We use the term TSDE as it was used in [2], which was the conference version of [1], and is also used in other variations of the algorithm [3]. The main result of [1] is to show that the Bayesian regret of TSDE accumulated up to time $T$ is bounded by $~ O (\sqrt{T})$ , where the $~ O (\cdot)$ notation hides constants and logarithmic factors. This result was derived under a technical assumption on the induced norm of the closed loop system. In this technical note, we present a minor variation of the TSDE algorithm and obtain a $~ O (\sqrt{T})$ bound on the Bayesian regret by imposing a much milder technical assumption, which is in terms of the spectral radius of the closed loop system (rather than the induced norm, as was assumed in [1]).

Ii Model and problem formulation

We consider the same model as [1]. For the sake of completeness, we present the model below.

Consider a linear quadratic system with state $x_{t} \in R^{n}$ , control input $u_{t} \in R^{m}$ , and disturbance $w_{t} \in R^{n}$ . We assume that the system starts from an initial state $x_{1} = 0$ and evolves over time according to

x_{t + 1} = A x_{t} + B u_{t} + w_{t}, t \geq 1,

(1)

where $A \in R^{n \times n}$ and $B \in R^{n \times m}$ are the system dynamics matrices. The noise ${w_{t}}_{t \geq 1}$ is an independent and identically distributed Gaussian process with $w_{t} \sim N (0, σ_{w}^{2} I)$ .

Remark 1

In [1], it was assumed that $σ_{w}^{2} = 1$ . Using a general $σ_{w}^{2} > 0$ does not fundamentally change any of the results or the proof arguments. $_{□}$

At each time $t$ , the system incurs a per-step cost given by

c(xt,ut)=x\raisebox0.0pt$⊺$tQxt+u\raisebox0.0pt$⊺$tRut,

(2)

where $Q$ and $R$ are positive definite matrices.

Let $θ\raisebox0.0pt$⊺$=[A,B]$ denote the parameters of the system. $θ \in R^{d \times n}$ , where $d = n + m$ . The performance of any policy $π = (π_{1}, π_{2}, \dots)$ is measured by the long-term average cost given by

J (π; θ) = limsup T \to \infty \frac{1}{T} E^{π} [T \sum t = 1 c (x_{t}, u_{t})] .

(3)

Let $J (θ)$ denote the minimum of $J (π; θ)$ over all policies. It is well known [4] that if the pair $(A, B)$ is stabilizable, then $J (θ)$ is given by

J (θ) = σ_{w}^{2} Tr (S (θ)),

where $S (θ)$ is the unique positive semi-definite solution of the following Riccati equation:

S(θ)=Q+A\raisebox0.0pt$⊺$S(θ)A−A\raisebox0.0pt$⊺$S(θ)B(R+B\raisebox0.0pt$⊺$S(θ)B)−1B\raisebox0.0pt$⊺$S(θ)A.

(4)

Furthermore, the optimal control policy is given by

u_{t} = G (θ) x_{t},

(5)

where the gain matrix $G (θ)$ is given by

G(θ)=−(R+B\raisebox0.0pt$⊺$S(θ)B)−1B\raisebox0.0pt$⊺$S(θ)A.

(6)

As in [1]

, we are interested in the setting where the system parameters are unknown. We denote the unknown parameters by a random variable

θ_{1}

and assume that there is a prior distribution on

θ_{1}

. The Bayesian regret of a policy

π

operating for horizon

T

is defined by

R (T; π) = E^{π} [T \sum t = 1 c (x_{t}, u_{t}) - T J (θ_{1})],

(7)

where the expectation is with respect to the prior on $θ_{1}$ , the noise processes, the initial conditions, and the potential randomizations done by the policy $π$ .

Iii Thomson sampling based learning algorithm

As in [1], we assume that the unknown model parameters $θ$ lie in a compact subset $Ω_{1}$ of $R^{d \times n}$ . We assume that there is a prior $μ_{1}$ on $Ω_{1}$ , which satisfies the following.

Assumption

There exist ${^θ}_{1} (i) \in R^{d}$ for $i \in {1, \dots, n}$ and a positive definite matrix $Σ_{1} \in R^{d \times d}$ such that for any $θ \in R^{d \times n}$ , $μ_{1} = {¯ μ}_{1} {∣ ∣}_{Ω_{1}}$ , where

{¯ μ}_{1} (θ) = n \prod i = 1 {¯ μ}_{1} (θ (i)) and {¯ μ}_{1} (θ (i)) = N ({^θ}_{1} (i), Σ_{1}) .

$_{■}$

We maintain a posterior distribution $μ_{t}$ on $Ω_{1}$ based on the history $(x_{1 : t - 1}, u_{1 : t - 1})$ of the observations until time $t$ . From standard results in linear Gaussian regression [5]

, we know that the posterior is a truncated Gaussian distribution

μ_{t} (θ) = [n \prod i = 1 {¯ μ}_{t} (θ (i))] {∣ ∣ ∣}_{Ω_{1}}

where ${¯ μ}_{t} (θ (i)) = N ({^θ}_{t} (i), Σ_{t})$ and ${^θt(i)}ni=1$ and $Σ_{t}$ can be updated recursively as follows:

	${^θ}_{t + 1} (i)$	$=^θt(i)+Σtzt(xt+1(i)−^θt(i)\raisebox0.0pt$⊺$zt)σ2w+z\raisebox0.0pt$⊺$tΣtzt,$		(8)
	$Σ_{t + 1}^{- 1}$	$=Σ−1t+1σ2wztz\raisebox0.0pt$⊺$t,$		(9)

where $zt=[x\raisebox0.0pt$⊺$t,u\raisebox0.0pt$⊺$t]\raisebox0.0pt$⊺$$ .

Iii-a Thompson sampling with dynamic episodes algorithm

We now present a minor variation of the Thompson sampling with dynamic episodes (TSDE) algorithm of [1]. As the name suggests, the algorithm operates in episodes of dynamic length. The key difference from [1] is that we enforce that each episode is of a minimum length $T_{min}$ . The choice of $T_{min}$ will be explained later.

Let $t_{k}$ and $T_{k}$ denote the start time and the length of episode $k$ , respectively. Episode $k$ has a minimum length of $T_{min}$ and ends when the length of the episode is strictly larger than the length of the previous episode (i.e., $t - t_{k} > T_{k - 1}$ ) or at the first time after $t_{k} + T_{min}$ when the determinant of the covariance $Σ_{t}$ falls below half of its value at time $t_{k}$ , i.e., $det Σ_{t} < \frac{1}{2} det Σ_{t_{k}}$ . Thus,

(10)

Note that the stopping condition (10) implies that

T_{min} + 1 \leq T_{k} \leq T_{k - 1} + 1, \forall k

(11)

If we select $T_{min} = 0$ in the above algorithm, we recover the stopping condition of [1].

The TSDE algorithm works as follows. At the beginning of episode $k$ , a parameter ${¯ θ}_{k}$ is sampled from the posterior distribution $μ_{t_{k}}$ . During the episode, the control inputs are generated using the sampled parameters ${¯ θ}_{k}$ , i.e.,

u_{t} = G ({¯ θ}_{k}) x_{t}, t_{k} \leq t < t_{k + 1} .

(12)

The complete algorithm is presented in Algorithm 1.

1:input:

Ω_{1}

{^θ}_{1}

Σ_{1}

2:initialization:

t \leftarrow 1

t_{0} \leftarrow - T_{min}

T_{- 1} \leftarrow T_{min}

k \leftarrow 0

3:for

t = 1, 2, \dots

4: observe

x_{t}

5: update

{¯ μ}_{t}

according to (8)–(9)

6: if

(t - t_{k} > T_{min})

and

((t - t_{k} > T_{k - 1}) or (det Σ_{t} < \frac{1}{2} det Σ_{t_{k}}))

8: then

T_{k} \leftarrow t - t_{k}

k \leftarrow k + 1

t_{k} \leftarrow t

10: sample

{¯ θ}_{k} \sim μ_{t}

11: end if

12: Apply control

u_{t} = G ({¯ θ}_{k}) x_{t}

13:end for

Algorithm 1 TSDE

Iii-B A technical assumption and the choice of minimum episode length

We make the following assumption on the support of the prior distribution.

Assumption

There exists a positive number $δ < 1$ such that for any $θ, ϕ \in Ω_{1}$ , where $θ\raisebox0.0pt$⊺$=[Aθ,Bθ]$ ,

ρ (A_{θ} + B_{θ} G (ϕ)) \leq δ .

$_{■}$

Assumption III-B is a weaker form of the following assumption imposed in [1] (since $ρ (A) \leq | | A | |$ for any matrix $A$ ).

Assumption

There exists a positive number $δ < 1$ such that for any $θ, ϕ \in Ω_{1}$ , where $θ\raisebox0.0pt$⊺$=[Aθ,Bθ]$ ,

∥ A_{θ} + B_{θ} G (ϕ) ∥ \leq δ .

$_{■}$

Note that it is much easier to satisfy Assumption III-B than Assumption III-B. For example, consider a family of matrices $A_{q} = [\begin{matrix} 1 - δ & q 0 & 1 - δ \end{matrix}]$ , where $q \in N$ and $0 < δ < 1$ . For each $q$ , the sprectral radius of $A_{q}$ is $1 - δ$ while its norm is at least $q$ . Thus, each $A_{q}$ satisfies Assumption III-B but not Assumption III-B.

Lemma 1

Assumption III-B implies that for any $ε \in (0, 1 - δ)$ , there exists an $α \geq 1$ such that for any $θ, ϕ \in Ω_{1}$ with $θ\raisebox0.0pt$⊺$=[Aθ,Bθ]$ and for any integer $t \geq 1$ ,

∥ (A_{θ} + B_{θ} G (ϕ))^{t} ∥ \leq α (ε + δ)^{t} .

$_{□}$

Proof

Let $L = {A_{θ} + B_{θ} G (ϕ) : θ, ϕ \in Θ}$ . Since $Θ$ is compact, so is $L$ . Now for any $L \in L$ , there exists a norm (call it ${norm}_{L}$ ) such that ${norm}_{L} (L) < ρ (L) + ε \leq δ + ε$ .

Since norms are continuous, there is an open ball centered at $L$ (let’s call this ${ball}_{L}$ ) such that for any $H \in {ball}_{L}$ , we have ${norm}_{L} (H) < δ + ε$ . Consider the collection of open balls ${{ball}_{L} : L \in L}$ . This is an open cover of compact set $L$ . So, there is a finite sub-cover. Let’s denote this sub-cover by ${ball}_{L_{1}}, \dots, {ball}_{L_{ℓ}}$ . By equivalence of norms, there is a finite constant $α_{k}$ such that $∥ A ∥ \leq α_{L_{k}} {norm}_{L_{k}} (A)$ for any matrix $A$ , for all $k \in {1, \dots, ℓ}$ . Let $α = max (1, {max}_{k} α_{L_{k}})$ .

Now consider an arbitrary $H \in L$ . It belongs to ${ball}_{L_{k}}$ for some $k \in {1, \dots, ℓ}$ . Therefore, ${norm}_{L_{k}} (H) < δ + ε$ . Hence, for any integer $t$ , the above inequalities and the submulitplicity of norms give that $∥ H^{t} ∥ \leq α_{L_{k}} {norm}_{L_{k}} (H^{t}) \leq α ({norm}_{L_{k}} (H))^{t} < α (δ + ε)^{t}$ . $_{■}$

A key implication of Lemma 1 is the following, which plays a critical role in analyzing the regret of TSDE.

Lemma 2

Fix an $ε \in (0, 1 - δ)$ and let $α$ be as given by Lemma 1. Define

(13)

Then, for $θ, ϕ \in Ω_{1}$ with $θ\raisebox0.0pt$⊺$=[Aθ,Bθ]$ and $τ \geq T_{min}^{*}$ , we have

∥ (A_{θ} + B_{θ} G (ϕ))^{τ} ∥ \leq 1

(14)

$_{□}$

Proof

The proof follows immediately from the choice of $T_{min}^{*}$ , Lemma 1 and (11). $_{■}$

Iii-C Regret bounds

The following result provides an upper bound on the regret of the proposed algorithm.

Theorem 1

Under Assumptions III and III-B and with $T_{min} \geq T_{min}^{*}$ , the regret of TSDE is upper bounded by

R (T; T S D E) \leq ~ O (σ_{w}^{2} (n + m) \sqrt{n T}) .

(15)

$_{□}$

The proof is presented in the next section.

Iv Regret analysis

For the ease of notation, we use $R (T)$ instead of $R (T; TSDE)$ in this section. Following the exact same steps as [1], we can show that

R (T) = R_{0} (T) + R_{1} (T) + R_{2} (T)

(16)

where

$R_{0} (T)$	$= E [K_{T} \sum k = 1 T_{k} J ({¯ θ}_{k})] - T E [J (θ_{1})],$	(17)
$R_{1} (T)$	$=E[KT∑k=1tk+1−1∑t=tk[x\raisebox0.0pt$⊺$tS(¯θk)xt−x\raisebox0.0pt$⊺$t+1S(¯θk)xt+1]]$	(18)
$R_{2} (T)$	$=E[KT∑k=1tk+1−1∑t=tk[(θ\raisebox0.0pt$⊺$1zt)\raisebox0.0pt$⊺$S(¯θk)θ\raisebox0.0pt$⊺$1zt$
	$−(θ\raisebox0.0pt$⊺$kzt)\raisebox0.0pt$⊺$S(¯θk)θ\raisebox0.0pt$⊺$kzt]]$	(19)

We establish the bound on $R (T)$ by individually bounding $R_{0} (T)$ , $R_{1} (T)$ , and $R_{2} (T)$ .

Lemma 3

The terms in (16) are bounded as follows:

$R_{0} (T) \leq ~ O (σ_{w}^{2} \sqrt{(n + m) T})$ .
$R_{1} (T) \leq ~ O (σ_{w}^{2} \sqrt{(n + m) T})$ .
$R_{2} (T) \leq ~ O (σ_{w}^{2} (n + m) \sqrt{n T})$ .

$_{□}$

Before presenting the proof of this lemma, we establish some preliminary results.

Iv-a Preliminary results

Let $X_{T} = {max}_{1 \leq t \leq T} ∥ x_{t} ∥$ denote the maximum of the norm of the state and $K_{T}$ denote the number of episodes until horizon $T$ .

Lemma 4

For any $q \geq 1$ and any $T \geq 1$ ,

E [X_{T}^{q}] \leq O (σ_{w}^{q} log T) .

$_{□}$

See Appendix A for proof.

Lemma 5

For any $q \geq 1$ , we have

E [X_{T}^{q} log X_{T}^{2}] \leq σ_{w}^{q} ~ O (1) .

$_{□}$

See Appendix B for proof.

Lemma 6

The number of episodes is bounded by

K_{T} \leq O (\sqrt{(n + m) T log (T X_{T}^{2})}) .

$_{□}$

See Appendix C for proof.

Remark 2

The statement of Lemmas 4 and 6 are the same as that of the corresponding lemmas in [1]. The proof of Lemma 4 in [1] relied on Assumption III-B. Since we impose a weaker assumption, our proof is more involved. The proof of Lemma 6 is similar to the proof of [1, Lemma 3]. However, since our TSDE algorithm is different from that in [1], some of the details of the proof are different. $_{□}$

Iv-B Proof of Lemma 3

We now prove each part of Lemma 3 separately.

Iv-B1 Proof of bound on $R_{0} (t)$

Following exactly the same argument as the proof of [1, Lemma 5], we can show that

R_{0} (T) \leq O (σ_{w}^{2} E [K_{T}]) .

(20)

Substituting the result of Lemma 6, we get

	$R_{0} (T)$	$\leq O (σ_{w}^{2} E [\sqrt{(n + m) T log (T X_{T}^{2})}])$
		$(a) \leq O (σ_{w}^{2} \sqrt{(n + m) T log (T E [X_{T}^{2}]}))$
		$(b) \leq ~ O (σ_{w}^{2} \sqrt{(n + m) T})$

where $(a)$ follows from Jensen’s inequality and $(b)$ follows from Lemma 4.

Iv-B2 Proof of bound on $R_{1} (t)$

Following exactly the same argument as in the proof of [1, Lemma 6], we can show that

R_{1} (T) \leq O (E [K_{T} X_{T}^{2}])

(21)

Substituting the result of Lemma 6, we get

R_{1} (T) \leq O (\sqrt{(n + m) T} E [X_{T}^{2} \sqrt{} log (T X_{T}^{2})])

(22)

Now, consider the term

$E [X_{T}^{2} \sqrt{log (T X_{T}^{2})}]$	$(a) \leq \sqrt{E [X_{T}^{4}] E [log (T X_{T}^{2})]}$
	$(b) \leq \sqrt{E [X_{T}^{4}] log (T E [X_{T}^{2}])}$
	$(c) \leq ~ O (σ_{w}^{2})$	(23)

where $(a)$ follows from Cauchy-Schwartz inequality, $(b)$ follows from Jensen’s inequality, and $(c)$ follows from Lemma 4.

Substituting (23) in (22), we get the bound on $R_{1} (T)$ .

Iv-B3 Proof of bound on $R_{2} (t)$

As in [1], we can bound the inner summand in $R_{2} (T)$ as

∥S(¯θk)0.5θ\raisebox0.0pt$⊺$1zt∥2−∥S(¯θk)0.5θ\raisebox0.0pt$⊺$kzt∥2≤O(XT∥(θ1−¯θk)\raisebox0.0pt$⊺$zt∥).

Therefore,

R2(T)≤O(E[XTKT∑k=1tk+1−1∑t=tk∥(θ1−¯θk)\raisebox0.0pt$⊺$zt∥]),

which is same as [1, Eq. (45)]. Now, by simplifying the term inside $O (\cdot)$ using Cauchy-Schwartz inequality, we get

		$E[XTKT∑k=1tk+1−1∑t=tk∥(θ1−¯θk)\raisebox0.0pt$⊺$zt∥]$
		$\leq \sqrt{E [K_{T} \sum k = 1 t_{k + 1} - 1 \sum t = t_{k} ∥ Σ_{t_{k}}^{- 0.5} (θ_{1} - {¯ θ}_{k}) ∥^{2}]}$
		$\times \sqrt{E [K_{T} \sum k = 1 t_{k + 1} - 1 \sum t = t_{k} X_{T}^{2} ∥ Σ_{t_{k}}^{0.5} z_{t} ∥^{2}]}$		(24)

Note that (24) is slightly different than the simplification of [1, Eq. (45)] using Cauchy-Schwartz inequality presented in [1, Eq. (46)], which used $Σ_{t}$ in each term in the right hand side instead of $Σ_{t_{k}}$ .

We bound each term of (24) separately as follows.

Lemma 7

We have the following inequality

		$E [K_{T} \sum k = 1 t_{k + 1} - 1 \sum t = t_{k} ∥ Σ_{t_{k}}^{- 0.5} (θ_{1} - {¯ θ}_{k}) ∥^{2}]$
		$\leq O (n (n + m) (T + E [K_{T}])) \leq O (n (n + m) T) .$

$_{□}$

See Appendix D for a proof.

Lemma 8

We have the following inequality

E [K_{T} \sum k = 1 t_{k + 1} - 1 \sum t = t_{k} X_{T}^{2} ∥ Σ_{t_{k}}^{0.5} z_{t} ∥^{2}] \leq ~ O ((n + m) σ_{w}^{4})

$_{□}$

See Appendix E for a proof.

We get the bound on $R_{2} (T)$ by substituting the result of Lemmas 7 and 8 in (24).

V Discussion and Conclusion

In this paper, we present a minor variation of the TSDE algorithm of [1] and show that its Bayesian regret up to time $T$ is bounded by $~ O (\sqrt{T})$ under a milder technical assumption that [1]. The result in [1] was derived under the assumption that there exists a $δ < 1$ such that for any $θ, ϕ \in Ω_{1}$ , $∥ A_{θ} + B_{θ} G (ϕ) ∥ \leq δ$ . We require that $ρ (A_{θ} + B_{θ} G (ϕ)) \leq δ$ . Our assumption on the sprectral radius of the closed loop system is milder and, in some sense, more natural than the assumption on the induced norm of the closed loop system.

The key technical result in [1] as well as our paper is Lemma 4, which shows that for any $q \geq 1$ , $E [X_{T}^{q}] \leq ~ O (σ_{w}^{2} log T)$ . The proof argument in both [1] as well as our paper is to show that there is some constant $α_{0}$ such that $X_{T} \leq α_{0} W_{T}$ . Under the stronger assumption in [1], one can show that for all $t$ , $∥ x_{t + 1} ∥ \leq δ ∥ x_{t} ∥ + ∥ w_{t} ∥$ , which directly implies that $X_{T} \leq W_{T} / (1 - δ)$ . Under the weaker assumption in this paper, the argument is more subtle. The basic intuition is that in each episode, the system is asymptotically stable and, being a linear system, also exponentially stable (in the sense of Lemma 1). So, if the episode length is sufficiently long, then we can ensure that $∥ x_{t_{k + 1}} ∥ \leq β ∥ x_{t_{k}} ∥ + ¯ α W_{T}$ , where $β < 1$ and $¯ α$ is a constant. This is sufficient to ensure that $X_{T} \leq α_{0} W_{T}$ for an appropriately defined $α_{0}$ .

The fact that each episode must be of length $T_{min}$ implies that the second triggering condition is not triggered for the first $T_{min}$ steps in an episode. Therefore, in this interval, the determinant of the covariance can be smaller than half of its value at the beginning of the episode. Consequently, we cannot use the same proof argument as [1] to bound $R_{2} (T)$ because that proof relied on the fact that for any $t \in {t_{k}, \dots, t_{k + 1} - 1}$ , $det Σ_{t}^{- 1} / det Σ_{t_{k}}^{- 1} \leq 2$ . So, we provide a variation of that proof argument, where we use a coarser bound on $det Σ_{t}^{- 1} / det Σ_{t_{k}}^{- 1}$ given by Lemma 10.

We conclude by observing that the milder technical assumption imposed in this paper may not be necessary. Numerical experiments indicate that the regret of the TSDE algorithm shows $~ O (\sqrt{T})$ behavior even when the uncertainty set $Ω_{1}$ does not satisfy Assumption III-B (as was also reported in [1]). This suggests that it might be possible to further relax Assumption III-B and still establish an $~ O (\sqrt{T})$ regret bound.

Appendix A Proof of Lemma 4

For the ease of notation, let $¯ δ = δ + ε$ , $¯ α = α / (1 - ¯ δ)$ , and $β = α {¯ δ}^{T_{min} + 1}$ . In addition, define $W_{T} = {max}_{1 \leq t \leq T} ∥ w_{t} ∥$ , ${¯ X}_{k} = {max}_{t_{k} < t \leq t_{k + 1}} ∥ x_{t} ∥$ , $Y_{k} = ∥ x_{t_{k}} ∥$ , and $H_{k} = A + B G ({¯ θ}_{k})$ where $A$ and $B$ are the true parameters.

From the system dynamics under the TSDE algorithm, we know that for any time $t \in {t_{k} + 1, \dots, t_{k + 1}}$ , we have

x_{t} = H_{k}^{t - t_{k}} x_{t_{k}} + t - 1 \sum j = t_{k} H_{k}^{t - 1 - j} w_{j}

Thus, from triangle inequality and Lemma 1, we get

	$∥ x_{t} ∥$	$\leq α {¯ δ}^{t - t_{k}} Y_{k} + [t - 1 \sum j = t_{k} α {¯ δ}^{t - 1 - j}] W_{T}$
		$≤α¯δt−tkYk+[α1−¯δ]\eqqcolon¯αWT.$		(25)

Now at time $t = t_{k + 1}$ , we have

	$Y_{k + 1}$	$= ∥ x_{t_{k + 1}} ∥ \leq α {¯ δ}^{T_{k}} Y_{k} + ¯ α W_{T} .$
		$\leq β Y_{k} + ¯ α W_{T}$		(26)

where the second inequality follows from (11), which implies $α¯δTk≤α¯δTmin+1\eqqcolonβ$ . Recursively expanding (26), we get

	$Y_{k}$	$\leq ¯ α W_{T} + β ¯ α W_{T} + \dots + β^{k - 2} ¯ α W_{T}$
		$≤¯α1−βWT\eqqcolon¯βWT.$		(27)

Substituting (27) is (25), we get that for any $t∈{tk+1,\allowbreak…,tk+1}$ , we have

∥xt||≤α¯δt−tk¯βWT+¯αWT≤[α¯β+¯α]\eqqcolonα0WT

where in the last inequality, we have used the fact that $¯ δ \in (0, 1)$ . Thus, for any episode $k$ , we have

{¯ X}_{k} = max t_{k} < t \leq t_{k + 1} ∥ x_{t} ∥ \leq α_{0} W_{T} .

Hence,

X_{T} \leq max {{¯ X}_{1}, \dots, {¯ X}_{K_{T}}} \leq α_{0} W_{T} .

Therefore, for any $q \geq 1$ , we have

E [X_{T}^{q}] \leq α_{0}^{q} E [W_{T}^{q}] = α_{0}^{q} E [max 1 \leq t \leq T ∥ w_{t} ∥^{q}]

(28)

From [1, Eq. (39)], we have that

E [max 1 \leq t \leq T ∥ w_{t} ∥^{q}] \leq σ_{w}^{q} O (log T) .

Substituting this is (28), we obtain the result of the lemma.

Appendix B Proof of Lemma 5

Since log is an increasing function, $log X_{T}^{2} \leq log max (e, X_{T}^{2})$ . Therefore,

	$E [X_{T}^{q} log X_{T}^{2}]$	$\leq E [X_{T}^{q} log max (e, X_{T}^{2})]$
		$\leq \sqrt{E [X_{T}^{2 q}] E [(log max (e, X_{T}^{2}))^{2}]}$		(29)

where the last inequality follows from Cauchy-Schwartz inequality. Since $(log x)^{2}$ is concave for $x \geq e$ , we can use Jensen’s inequality to write

$E [(log max (e, X_{T}^{2}))^{2}]$	$\leq (log (E [max (e, X_{T}^{2})]))^{2}$
	$\leq (log (e + E [X_{T}^{2}]))^{2}$
	$(a) \leq (log (e + O (σ_{w}^{2} log T)))^{2}$
	$\leq ~ O (1)$	(30)

where $(a)$ uses Lemma 4. Substituting (30) in (29) and using Lemma 4 for bounding $E [X_{T}^{2 q}]$ , we get

	$E [X_{T}^{q} log X_{T}^{2}]$	$\leq \sqrt{E [X_{T}^{2 q}] E [(log max (e, X_{T}^{2}))^{2}]}$
		$\leq σ_{w}^{q} ~ O (1) .$

Appendix C Proof of Lemma 6

The high-level idea of the proof is same as that of [1, Lemma 3]. Define macro episodes with start times $t_{n_{i}}$ , $i \in N_{> 0}$ , where $n_{1} = 1$ and for $i \geq 1$ ,

n_{i + 1} = min {k > n_{i} ∣ ∣ det Σ_{t_{k}} < \frac{1}{2} det Σ_{t_{k - 1}}} .

Thus, a new macro-episode starts whenever an episode ends due to the second stopping criterion. Let $M$ denote the number of macro-episodes until time $T$ and define $n_{M + 1} = K_{T} + 1$ . Let ${¯ T}_{i}$ denote the length of the $i$ -th macro-episode. Within a macro-episode, all but the last episode must be triggered by the first stopping criterion. Thus, for $k \in {n_{i}, n_{i} + 1, \dots, n_{i + 1} - 2}$ ,

T_{k} = max {T_{k - 1} + 1, T_{min} + 1} = T_{k - 1} + 1

where the last equality follows from (11). Hence, by following exactly the same argument as [1], we have

n_{i + 1} - n_{i} \leq \sqrt{2 {¯ T}_{i}}

and therefore following [1, Eq. (40)], we have

K_{T} \leq \sqrt{2 M T}

(31)

which is same as [1, Eq. (41)].

Now, observe that

	$det Σ_{T}^{- 1}$	$(a) \geq det Σ_{t_{n_{M}}}^{- 1} (b) \geq 2 det Σ_{t_{n_{M - 1}}}^{- 1}$
		$\geq \dots \geq 2^{M - 1} det Σ_{1}^{- 1},$		(32)

where $(a)$ follows because ${det Σ_{t}^{- 1}}_{t \geq 1}$ is a non-decreasing sequence (because $Σ_{1}^{- 1} \leq Σ_{2}^{- 1} \dots$ ) and $(b)$ and subsequent inequalities follow from the definition of the macro episode and the second triggering condition.

Then following the same idea as the rest of the proof in [1], we get

M \leq O ((n + m) log (T X_{T}^{2})) .

(33)

Substituting (33) in (31), we obtain the result of the lemma.

Appendix D Proof of Lemma 7

Observe that the summand is constant for each episode. Therefore,

		$E[KT∑k=1tk+1−1∑t=tk[∥Σ−0.5tk(θ1−¯θk)\raisebox0.0pt$⊺$∥2]]$
		$=E[KT∑k=1[Tk∥Σ−0.5tk(θ1−¯θk)\raisebox0.0pt$⊺$∥2]]$
		$(a)≤E[KT∑k=1[(Tk−1+1)∥Σ−0.5tk(θ1−¯θk)\raisebox0.0pt$⊺$∥2]]$
		$=∞∑k=1E[1{tk≤T}(Tk−1+1)∥Σ−0.5tk(θ1−¯θk)\raisebox0.0pt$⊺$∥2]$
		$=∞∑k=1E[E[1{tk≤T}(Tk−1+1)∥Σ−0.5tk(θ1−¯θk)\raisebox0.0pt$⊺$∥2∣∣htk]]$
		$($

A relaxed technical assumption for posterior sampling-based reinforcement learning for control of unknown linear systems

Related Research

Logarithmic Regret for Reinforcement Learning with Linear Function Approximation

Posterior Sampling for Reinforcement Learning Without Episodes

Learning Zero-sum Stochastic Games with Posterior Sampling

Scalable regret for learning to control network-coupled subsystems with unknown dynamics

On learning Whittle index policy for restless bandits with scalable regret

Making Non-Stochastic Control (Almost) as Easy as Stochastic

Damped Online Newton Step for Portfolio Selection

I Introduction

Ii Model and problem formulation

Remark 1

Iii Thomson sampling based learning algorithm

Assumption

Iii-a Thompson sampling with dynamic episodes algorithm

Iii-B A technical assumption and the choice of minimum episode length

Assumption

Assumption

Lemma 1

Proof

Lemma 2

Proof

Iii-C Regret bounds

Theorem 1

Iv Regret analysis

Lemma 3

Iv-a Preliminary results

Lemma 4

Lemma 5

Lemma 6

Remark 2

Iv-B Proof of Lemma 3

Iv-B1 Proof of bound on R0(t)

Iv-B2 Proof of bound on R1(t)

Iv-B3 Proof of bound on R2(t)

Lemma 7

Lemma 8

V Discussion and Conclusion

Appendix A Proof of Lemma 4

Appendix B Proof of Lemma 5

Appendix C Proof of Lemma 6

Appendix D Proof of Lemma 7

Iv-B1 Proof of bound on $R_{0} (t)$

Iv-B2 Proof of bound on $R_{1} (t)$

Iv-B3 Proof of bound on $R_{2} (t)$