On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator

1 Introduction

Imitation learning is a paradigm that learns from expert demonstration to perform a task. The most straightforward approach of imitation learning is behavioral cloning (Pomerleau, 1991), which learns from expert trajectories to predict the expert action at any state. Despite its simplicity, behavioral cloning ignores the accumulation of prediction error over time. Consequently, although the learned policy closely resembles the expert policy at a given point in time, their trajectories may diverge in the long term.

To remedy the issue of error accumulation, inverse reinforcement learning (Russell, 1998; Ng and Russell, 2000; Abbeel and Ng, 2004; Ratliff et al., 2006; Ziebart et al., 2008; Ho and Ermon, 2016) jointly learns a reward function and the corresponding optimal policy, such that the expected cumulative reward of the learned policy closely resembles that of the expert policy. In particular, as a unifying framework of inverse reinforcement learning, generative adversarial imitation learning (GAIL) (Ho and Ermon, 2016) casts most existing approaches as iterative methods that alternate between (i) minimizing the discrepancy in expected cumulative reward between the expert policy and the policy of interest and (ii) maximizing such a discrepancy over the reward function of interest. Such a minimax optimization formulation of inverse reinforcement learning mirrors the training of generative adversarial networks (GAN), which alternates between updating the generator and discriminator, respectively.

Despite its prevalence, inverse reinforcement learning, especially GAIL, is notoriously unstable in practice. More specifically, most inverse reinforcement learning approaches involve (partially) solving a reinforcement learning problem in an inner loop, which is often unstable, especially when the intermediate reward function obtained from the outer loop is ill-behaved. This is particularly the case for GAIL, which, for the sake of computational efficiency, alternates between policy optimization and reward function optimization without fully solving each of them. Moreover, such instability is exacerbated when the policy and reward function are both parameterized by deep neural networks. In this regard, the training of GAIL is generally more unstable than that of GAN, since policy optimization in deep reinforcement learning is often more challenging than training a standalone deep neural network.

In this paper, we take a first step towards theoretically understanding and algorithmically taming the instability in imitation learning. In particular, under a minimax optimization framework, we for the first time establish the global convergence of GAIL under a fundamental setting known as linear quadratic regulators (LQR). Such a setting of LQR is studied in a line of recent works (Bradtke, 1993; Fazel et al., 2018; Tu and Recht, 2017, 2018; Dean et al., 2018a, b; Simchowitz et al., 2018; Dean et al., 2017; Hardt et al., 2018) as a lens for theoretically understanding more general settings in reinforcement learning. See Recht (2018) for a thorough review. In imitation learning, particularly GAIL, the setting of LQR captures four critical challenges of more general settings:

the minimax optimization formulation,
the lack of convex-concave geometry,
the alternating update of policy and reward function, and
the instability of the dynamical system induced by the intermediate policy and reward function (which differs from the aforementioned algorithmic instability).

Under such a fundamental setting, we establish a global sublinear rate of convergence towards a saddle point of the minimax optimization problem, which is guaranteed to be unique and recovers the globally optimal policy and reward function. Moreover, we establish a local linear rate of convergence, which, combined with the global sublinear rate of convergence, implies a global Q-linear rate of convergence. A byproduct of our theory is the stability of all the dynamical systems induced by the intermediate policies and reward functions along the solution path, which addresses the key challenge in (iv) and plays a vital role in our analysis. At the core of our analysis is a new potential function tailored towards non-convex-concave minimax optimization with alternating update, which is of independent interest. To ensure the decay of potential function, we rely on the aforementioned stability of intermediate dynamical systems along the solution path. To achieve such stability, we unveil an intriguing “self-enforcing” stabilizing mechanism, that is, with a proper configuration of stepsizes, the solution path approaches the critical threshold that separates stable and unstable regimes at a slower rate as it gets closer to such a threshold. In other words, such a threshold forms an implicit barrier, which ensures the stability of the intermediate dynamical systems along the solution path without any explicit regularization.

Our work extends the recent line of works on reinforcement learning under the setting of LQR (Bradtke, 1993; Recht, 2018; Fazel et al., 2018; Tu and Recht, 2017, 2018; Dean et al., 2018a, b; Simchowitz et al., 2018; Dean et al., 2017; Hardt et al., 2018) to imitation learning. In particular, our analysis relies on several geometric lemmas established in Fazel et al. (2018), which are listed in §F for completeness. However, unlike policy optimization in reinforcement learning, which involves solving a minimization problem where the objective function itself serves as a good potential function, imitation learning involves solving a minimax optimization problem, which requires incorporating the gradient into the potential function. In particular, the stability argument developed in Fazel et al. (2018), which is based on the monotonicity of objective functions along the solution path, is no longer applicable, as minimax optimization alternatively decreases and increases the objective function at each iteration. In a broader context, our work takes a first step towards extending the recent line of works on nonconvex optimization, e.g., Baldi and Hornik (1989); Du and Lee (2018); Wang et al. (2014a, b); Zhao et al. (2015); Ge et al. (2015, 2017a, 2017b); Anandkumar et al. (2014); Bandeira et al. (2016); Li et al. (2016a, b); Hajinezhad et al. (2016); Bhojanapalli et al. (2016); Sun et al. (2015, 2018), to non-convex-concave minimax optimization (Du and Hu, 2018; Sanjabi et al., 2018; Rafique et al., 2018; Lin et al., 2018; Dai et al., 2017, 2018a, 2018b; Lu et al., 2019) with alternating update, which is prevalent in reinforcement learning, imitation learning, and generative adversarial learning, and poses significantly more challenges.

In the rest of this paper, §2 introduces imitation learning, the setting of LQR, and the generative adversarial learning framework. In §3, we introduce the minimax optimization formulation and the gradient algorithm. In §4 and §5, we present the theoretical results and sketch the proof. We defer the detailed proof to §A-§F of the appendix.

Notation. We denote by $∥ \cdot ∥$ the spectral norm and $∥ \cdot ∥_{F}$

the Frobenius norm of a matrix. For vectors, we denote by

∥ \cdot ∥_{2}

the Euclidean norm. In this paper, we write parameters in the matrix form, and correspondingly, all the Lipschitz conditions are defined in the Frobenius norm.

2 Background

In the following, we briefly introduce the setting of LQR in §2.1 and imitation learning in §2.2. To unify the notation of LQR and more general reinforcement learning, we stick to the notion of cost function instead of reward function throughout the rest of this paper.

2.1 Linear Quadratic Regulator

In reinforcement learning, we consider a Markov decision process

{X,\cU,c,T,D0}

, where an agent interacts with the environment in the following manner. At the

t

-th time step, the agent selects an action

ut∈\cU

based on its current state

x_{t} \in X

, and the environment responds with the cost

c_{t} = c (x_{t}, u_{t})

and the next state

x_{t + 1} \in X

, which follows the transition dynamics

T

. Our goal is to find a policy

u_{t} = π_{t} (x_{t})

that minimizes the expected cumulative cost. In the setting of LQR, we consider

X = R^{d}

and

\cU=Rk

. The dynamics and cost function take the form

x_{t + 1} = A x_{t} + B u_{t}, c (x_{t}, u_{t}) = x_{t}^{⊤} Q x_{t} + u_{t}^{⊤} R u_{t},

where $A \in R^{d \times d}$ , $B \in R^{d \times k}$ , $Q \in R^{d \times d}$ , and $R \in R^{k \times k}$ with $Q, R ≻ 0$ . The problem of minimizing the expected cumulative cost is then formulated as the optimization problem

	$\minimizeπt$	$E [\sum_{t = 0}^{\infty} x_{t}^{⊤} Q x_{t} + u_{t}^{⊤} R u_{t}]$		(1)
	subject to	$x_{t + 1} = A x_{t} + B u_{t}, u_{t} = π_{t} (x_{t}), x_{0} \sim D_{0},$

where $D_{0}$ is a given initial distribution. Here we consider the infinite-horizon setting with a stochastic initial state $x_{0} \sim D_{0}$ . In this setting, the optimal policy $π_{t}$ is known to be static and takes the form of linear feedback $π_{t} (x_{t}) = - K x_{t}$ , where $K \in R^{k \times d}$ does not depend on $t$ (Anderson and Moore, 2007). Throughout the rest of this paper, we also refer to $K$ as policy and drop the subscript $t$ in $π_{t}$ . To ensure the expected cumulative cost is finite, we require the spectral radius of $(A - B K)$ to be less than one, which ensures that the dynamical system

x_{t + 1} = A x_{t} + B u_{t} = (A - B K) x_{t}

(2)

is stable. For a given policy $K$ , we denote by $C (K; Q, R)$ the expected cumulative cost in (1). For notational simplicity, we define

Σ_{K} = E [\sum_{t = 0}^{\infty} x_{t} x_{t}^{⊤} | π_{t} (x_{t}) = - K x_{t}], Σ_{0} = E [x_{0} x_{0}^{⊤}] .

(3)

By (3), we have the following equivalent form of $C (K; Q, R)$

C (K; Q, R) = Tr (Σ_{K} Q) + Tr (K Σ_{K} K^{⊤} R) = ⟨ Σ_{K}, Q ⟩ + ⟨ K Σ_{K} K^{⊤}, R ⟩,

(4)

where $⟨ \cdot, \cdot ⟩$ denotes the matrix inner product. Also, throughout the rest of this paper, we assume that the initial distribution $D_{0}$ satisfies $σ_{min} (Σ_{0}) > 0$ . See Recht (2018) for a thorough review of reinforcement learning in the setting of LQR.

2.2 Imitation Learning

In imitation learning, we parameterize the cost function of interest by $c (x_{t}, u_{t}; θ)$ , where $θ$ denotes the unknown cost parameter. In the setting of LQR, we have $θ = (Q, R)$ . We observe expert trajectories in the form of ${(x_{t}, u_{t}, c_{t})}_{t = 0}^{\infty}$ , which are induced by the expert policy $π_{E}$ . As a unifying framework of inverse reinforcement learning, GAIL (Ho and Ermon, 2016) casts max-entropy inverse reinforcement learning (Ziebart et al., 2008) and its extensions as the following minimax optimization problem

	$max θ min π {E [\sum_{t = 0}^{\infty} c_{t} (x_{t}, u_{t}; θ) \| u_{t} = π (x_{t})] - H (π)$
	$- E [\sum_{t = 0}^{\infty} c_{t} (x_{t}, u_{t}; θ) \| u_{t} = π_{E} (x_{t})] - ψ (θ)},$		(5)

where for ease of presentation, we restrict to deterministic policies in the form of $u_{t} = π (x_{t})$ . Here $H (π)$ denotes the causal entropy of the dynamical system ${x_{t}}_{t = 0}^{\infty}$ induced by $π$ , which takes value zero in our setting of LQR, since the transition dynamics in (2) is deterministic conditioning on $x_{t}$ . Meanwhile, $ψ (θ)$ is a regularizer on the cost parameter.

The minimax optimization formulation in (2.2) mirrors the training of GAN (Goodfellow et al., 2014), which seeks to find a generator of distribution that recovers a target distribution. In the training of GAN, the generator and discriminator are trained simultaneously, in the manner that the discriminator maximizes the discrepancy between the generated and target distributions, while the generator minimizes such a discrepancy. Analogously, in imitation learning, the policy $π$ of interest acts as the generator of trajectories, while the expert trajectories act as the target distribution. Meanwhile, the cost parameter $θ$ of interest acts as the discriminator, which differentiates between the trajectories generated by $π$ and $π_{E}$ . Intuitively, maximizing over the cost parameter $θ$ amounts to assigning high costs to the state-action pairs visited more by $π$ than $π_{E}$ . Minimizing over $π$ aims at making such an adversarial assignment of cost impossible, which amounts to making the visitation distributions of $π$ and $π_{E}$ indistinguishable.

3 Algorithm

In the sequel, we first introduce the minimax formulation of generative adversarial imitation learning in §3.1, then we present the gradient algorithm in §3.2.

3.1 Minimax Formulation

We consider the minimax optimization formulation of the imitation learning problem,

min K \in K max θ \in Θ m (K, θ), where m (K, θ) = C (K; θ) - C (K_{E}; θ) - ψ (θ) .

(6)

Here we denote by $θ = (Q, R)$ the cost parameter, where $Q \in R^{d \times d}$ and $R \in R^{k \times k}$ are both positive definite matrices, and $Θ$ is the feasible set of cost parameters. We assume $Θ$ is convex and there exist positive constants $α_{Q}$ , $α_{R}$ , $β_{Q}$ , and $β_{R}$ such that for any $(Q, R) \in Θ$ , it holds that

α_{Q} I ⪯ Q ⪯ β_{Q} I, α_{R} I ⪯ R ⪯ β_{R} I .

(7)

Also, $K$ consists of all stabilizing policies, such that $ρ (A - B K) < 1$ for all $K \in K$ , where $ρ$

is the spectral radius defined as the largest complex norm of the eigenvalues of a matrix. The expert policy is defined as

KE=\argminKC(K;~θ)

for an unknown cost parameter

~ θ \in Θ

. However, note that

~ θ

is not necessarily the unique cost parameter such that

K_{E}

is optimal. Hence, our goal is to find one of such cost parameters

θ^{*}

that

K_{E}

is optimal. The term

ψ (\cdot)

is the regularizer on the cost parameter, which is set to be

γ

-strongly convex and

ν

-smooth.

To understand the minimax optimization problem in (6), we first consider the simplest case with $ψ (θ) \equiv 0$ . A saddle point $(K^{*}, θ^{*})$ of the objective function in (6), defined by

m (K^{*}, θ^{*}) = min K m (K, θ^{*}) = max θ m (K^{*}, θ),

(8)

has the following desired properties. First, we have that the optimal policy $K^{*}$ recovers the expert policy $K_{E}$ . By the optimality condition in (8), we have

C (K^{*}; ~ θ) - C (K_{E}; ~ θ) = m (K^{*}; ~ θ) \leq m (K^{*}; θ^{*}) = C (K^{*}; θ^{*}) - C (K_{E}; θ^{*}) \leq 0,

(9)

where the first inequality follows from the optimality of $θ^{*}$ and the second inequality follows from the optimality of $K^{*}$ . Since the optimal solution to the policy optimization problem ${min}_{K} C (K; ~ θ)$ is unique (as proved in §5), we obtain from (9) that $K^{*} = K_{E}$ . Second, $K_{E}$ is an optimal policy with respect to the cost parameter $θ^{*}$ , since by $K^{*} = K_{E}$ and the optimality condition $K∗=\argminKC(K;θ∗)$ , we have $KE=\argminKC(K;θ∗)$ . In this sense, the saddle point $(K^{*}; θ^{*})$ of (6) recovers a desired cost parameter and the corresponding optimal policy.

Although $ψ (\cdot) \equiv 0$ brings us desired properties of the saddle point, there are several reasons we can not simply discard this regularizer. The first reason is that a strongly convex regularizer improves the geometry of the problem and makes the saddle point of (6) unique, which eliminates the ambiguity in learning the desired cost parameter. Second, the regularizer draws connection to the existing optimization formulations of GAN. For example, as shown in Ho and Ermon (2016), with a specific choice of $ψ (\cdot)$ , (6) reduces to the classical optimization formulation of GAN (Goodfellow et al., 2014), which minimizes the Jensen-Shannon divergence bewteen the generator and target distributions.

3.2 Gradient Algorithm

To solve the minimax optimization problem in (6), we consider the alternating gradient updating scheme,

	$K_{i + 1}$	$\leftarrow K_{i} - η \cdot \nabla_{K} m (K_{i}, θ_{i}) = K_{i} - η \cdot \nabla_{K} C (K_{i}; θ_{i}),$		(10)
	$θ_{i + 1}$	$\leftarrow Π_{Θ} [θ_{i} + λ \cdot (\nabla_{Q} m (K_{i + 1}, θ_{i}), \nabla_{R} m (K_{i + 1}, θ_{i}))] .$		(11)

Here $Π_{Θ} [\cdot]$ is the projection operator onto the convex set $Θ$ , which ensures that each iterate $θ_{i}$ stays within $Θ$ .

There are several ways to obatin the gradient in (10) without knowing the dynamics $X_{t + 1} = A x_{t} + B u_{t}$ but based on the trajectory ${(x_{t}, u_{t}, c_{t})}_{t = 0}^{\infty}$ . One example is the deterministic policy gradient algorithm (Silver et al., 2014). In specific, the gradient of the cost function is obtained through the limit

lim σ ↓ 0 E [\nabla_{K} π_{K, σ} (u | x) \cdot Q^{π_{K, σ}} (x, u)],

where $π_{K, σ} (u | x)$ is a stochastic policy that takes the form $u | x \sim N (- K x, σ^{2} I)$ . Here $Q^{π_{K, σ}} (x, u)$ is the action-value function associated with the policy $π_{K, σ} (u | x)$ , defined as the expected total cost of the policy $π_{K, σ} (u | x)$ starting at state $x$ and action $u$

, which can be estimated based on the trajectory

{(x_{t}, u_{t}, c_{t})}_{t = 0}^{\infty}

. An alternative approach is the evolutionary strategy (Salimans et al., 2017), which uses zeroth-order information to approximate

\nabla_{K} C (K; θ)

with a random perturbation,

E_{vec (ε) \sim N (0, σ^{2} I)} [C (K_{i} + ε; θ_{i}) \cdot ε] / σ^{2},

where $ε \in R^{k \times d}$

is a random matrix in

R^{k \times d}

with a sufficiently small variance

σ^{2}

To obtain the gradient in (11), we have

	$\nabla_{Q} m (K_{i + 1}; θ_{i})$	$= Σ_{K_{i + 1}} - Σ_{K_{E}} - \nabla_{Q} ψ (Q_{i}, R_{i}),$		(12)
	$\nabla_{R} m (K_{i + 1}; θ_{i})$	$= K_{i + 1} Σ_{K_{i + 1}} K_{i + 1}^{⊤} - K_{E} Σ_{K_{E}} K_{E}^{⊤} - \nabla_{R} ψ (Q_{i}, R_{i}) .$		(13)

Here $Σ_{K} = E [\sum_{t = 0}^{\infty} x_{t} x_{t}^{⊤}]$ with ${x_{t}}_{t = 0}^{\infty}$ generated by policy $K$ , which can be estimated based on the trajectory ${(x_{t}, u_{t}, c_{t})}_{t = 0}^{\infty}$ .

4 Main Results

In this section, we present the convergence analysis of the gradient algorithm in (10) and (11). We first prove that the solution path ${K_{i}}_{i \geq 0}$ are guaranteed to be stabilizing and then establish the global convergence. For notational simplicity, we define the following constants,

α = min {α_{Q}, α_{R}}, σ_{θ} = sup (Q, R) \in Θ (∥ Q ∥_{F}^{2} + ∥ R ∥_{F}^{2})^{1 / 2}, μ = σ_{min} (Σ_{0}) > 0,

(14)

where $α_{Q}$ and $α_{R}$ are defined in (7), and $Σ0=\EE[x0x⊤0]$ . Also, we define

	$M$	$= β_{Q} \cdot Tr (Σ_{K_{0}}) + β_{R} \cdot % Tr (K_{0} Σ_{K_{0}} {K_{0}}^{⊤}),$		(15)
	$F$	$= max {∥ Σ_{K_{E}} ∥_{F} + sup (Q, R) \in Θ ∥ \nabla_{Q} ψ (Q, R) ∥_{F}, ∥ K_{E} Σ_{K_{% E}} K_{E}^{⊤} ∥_{F} + sup (Q, R) \in Θ ∥ \nabla_{R} ψ (Q, R) ∥_{F}},$		(16)

which play a key role in upper bounding the cost function $C (K; θ)$ .

4.1 Stability Guarantee

A minimum requirement in reinforcement learning is to obtain a stabilizing policy such that the dynamical system does not tend to infinity. Throughout this paper, we employ a notion of uniform stability, which states that there exists a constant $S$ such that $∥ Σ_{K_{i}} ∥ \leq S$ for all $i$ . Moreover, the uniform stability also allows us to establish the smoothness of $m (K, θ)$ , which is discussed in §5.2.

Recall that we assume $Q$ and $R$ are positive definite. Therefore, the uniform stability is implied by the boundedness of the cost function $C (K_{i}; θ_{i})$ , since we have

α_{Q} \cdot ∥ Σ_{K_{i}} ∥ \leq α_{Q} \cdot Tr (Σ_{K_{i}}) \leq ⟨ Σ_{K_{i}}, Q_{i} ⟩ \leq C (K_{i}; θ_{i}),

(17)

where the second inequality follows from the properties of trace and the assumption $Q ⪰ α_{Q} I$ in (7). However, it remains difficult to show that the cost function $C (K_{i}; θ_{i})$ is upper bounded. Although the update of policy in (10) decreases the cost function $C (K_{i}; θ_{i})$ , the update of cost parameter increases $m (K_{i}; θ_{i})$ , which possibly increases the cost function $C (K_{i}; θ_{i})$ . To this end, we choose suitable stepsizes as in the next condition to ensure the boundedness of the cost function.

For the update of policy and cost parameter in (10) and (11), let

η \leq min {

\frac{α_{Q}^{3} μ^{5 / 2} (α F + 2 M)^{- 7 / 2}}{16 κ_{1}^{1 / 2} κ_{2} \cdot ∥ B ∥}, \frac{α_{Q}}{32 κ_{1} (α F + 2 M)}, \frac{2 M}{α_{Q} α_{R} μ^{2}}}, λ / η \leq \frac{α_{Q} α_{R} α^{2} μ^{2}}{2 M (α F + 2 M)} .

Here $α$ , $μ$ , $F$ , and $M$ are defined in (14), (15), and (16). The constants $κ_{1}$ and $κ_{2}$ are defined as

κ_{1} = β_{R} + (α F + 2 M) \cdot ∥ B ∥^{2} / μ, κ_{2} = 1 + (μ α_{Q})^{- 1 / 2} (α F + 2 M)^{1 / 2} .

(18)

The next lemma shows that the solution path ${K_{i}}_{i \geq 0}$ is uniformly stabilizing, and meanwhile, along the solution path, the cost function $C (K_{i}; θ_{i})$ and $∥ K_{i} ∥^{2}$ are both upper bounded.

Under Condition 4.1, we have

C (K_{i}; θ_{i}) \leq α F + 2 M, ∥ K_{i} ∥^{2} \leq (α F + 2 M) / (α_{R} μ), ∥ Σ_{K_{i}} ∥ \leq (α F + 2 M) / α_{Q}

for all $i \geq 0$ .

Proof.

See §A for a detailed proof. ∎

4.2 Global Convergence

Before showing the gradient algorithm converges to the saddle point $(K^{*}, θ^{*})$ of (6), we establish its uniqueness. We define the proximal gradient of the objective function $m (K, θ)$ in (6) as

L (K, θ) = (\nabla_{K} m (K, θ), θ - Π_{Θ} [θ - \nabla_{θ} m (K, θ)]) .

(19)

Then a proximal stationary point is defined by $L (K, θ) = 0$ . [Uniqueness of Saddle Point] There exists a unique proximal stationary point, denoted as $(K^{*}, θ^{*})$ , of the objective function $m (K, θ)$ in (6), which is also its unique saddle point.

Proof.

See §D.1 for a detailed proof. ∎

To analyze the convergence of the gradient algorithm, we first need to establish the Lipschitz continuity and smoothness of $m (K, θ)$ . However, the cost function $C (K; θ)$ becomes steep as the policy $K$ is close to unstabilizing. Therefore, we do not have such desired Lipschitz continuity and smoothness of $Σ_{K}$ and $K Σ_{K} K^{⊤}$ with respect to $K$ . However, given $∥ Σ_{K} ∥$ is upper bounded as in Lemma 4.1, we obtain such desired properties in the following lemma.

For notational simplicity, we slightly abuse the notation and rewrite $θ$ as a block diagonal matrix and correspondingly define $V (K)$ ,

θ = (\begin{matrix} Q & 0 0 & R \end{matrix}) \in R^{(d + k) \times (d + k)}, V (K) = (\begin{matrix} Σ_{K} & 0 0 & K Σ_{K} K^{⊤} \end{matrix}) \in R^{(d + k) \times (d + k)} .

(20)

Then the objective function takes the form

(21)

We assume that the initial policy $K_{0}$ of the gradient algorithm is stabilizing. Under Condition 4.1, there exists a compact set $K^{†} ⫋ K$ such that $K_{i} \in K^{†}$ for all $i \geq 0$ . Also, there exist constants $τ_{V}$ and $ν_{V}$ such that the matrix-valued function $V (K)$ defined in (20) is $τ_{V}$ -Lipschitz continuous and $ν_{V}$ -smooth over $K^{†}$ . That is, for any $K_{1}, K_{2} \in K^{†}$ and $j, ℓ \in [d + k]$ , we have

∥ V (K_{1}) - V (K_{2}) ∥_{F} \leq τ_{V} \cdot ∥ K_{1} - K_{2} ∥_{F}, ∥ \nabla V_{j, ℓ} (K_{1}) - \nabla V_{j, ℓ} (K_{2}) ∥_{F} \leq ν_{V} / (d + k) \cdot ∥ K_{1} - K_{2} ∥_{F} .

Proof.

See §5.2 for a detailed proof. ∎

Note that the cost parameter $(Q, R)$ is only identifiable up to a multiplicative constant. Recall that we assume $α_{Q} I ⪯ Q ⪯ β_{Q} I$ and $α_{R} I ⪯ R ⪯ β_{R} I$ . In the sequel, we establish the sublinear rate of convergence with a proper choice of $α_{Q}$ , $α_{R}$ , $β_{Q}$ , and $β_{R}$ , which is characterized by the following condition.

We assume that $α_{Q}$ , $α_{R}$ , $β_{Q}$ , and $β_{R}$ satisfy

α_{Q} α_{R} α^{2} γ \geq 14 σ_{θ} ν_{V} M (α F + 2 M),

(22)

where $F$ , $M$ , and $σ_{θ}$ are defined in (14), (15) and (16), and $ν_{V}$ is defined in Lemma 4.2.

The following condition, together with Condition 4.1, specifies the required stepsizes to establish the global convergence of the gradient algorithm.

For the stepsizes $η$ and $λ$ in (10) and (11), let

η \leq min {\frac{1}{100 τ_{V}}, \frac{1}{2 σ_{θ} ν_{V}}}, λ \leq min {\frac{1}{100 (τ_{V} + ν)}, \frac{3 ν_{V} σ_{θ}}{100 τ_{V}^{2}}, \frac{γ}{100 ν^{2}}}, η / λ < \frac{γ}{7 ν_{V} σ_{θ}} .

In the following, we establish the global convergence of the gradient algorithm. Recall that as defined in (19), $L (K, θ)$ is the proximal gradient of the objective function $m (K, θ)$ defined in (6).

Under Conditions 4.1, 4.2, and 4.2, we have ${lim}_{i \to \infty} ∥ L (K_{i}, θ_{i}) ∥_{F} = 0$ , which implies that ${(K_{i}, θ_{i})}_{i = 0}^{\infty}$ converges to the unique saddle point $(K^{*}, θ^{*})$ of $m (K, θ)$ . To characterize the rate of convergence, we define $Γ (ε)$ as the smallest iteration index that $∥ L (K_{i}, θ_{i}) ∥_{F}^{2}$ is below an error $ε > 0$ ,

Γ (ε) = min {i | ∥ L (K_{i}, θ_{i}) ∥_{F}^{2} \leq ε} .

(23)

Then there exists a constant $ζ$ , which depends on $K_{0}$ , $θ_{0}$ , $η$ , and $λ$ (as specified in (37)), such that $Γ (ε) \leq ζ / ε$ for any $ε$ .

Proof.

See §5.2 for a detailed proof. ∎

To understand Condition 4.2, we consider a simple case where the regularizer $ψ (\cdot)$ is the squared penalty centered at some point $(¯ Q, ¯ R) \in Θ$ , that is,

ψ (Q, R) = γ \cdot (∥ Q - ¯ Q ∥_{F}^{2} + ∥ R - ¯ R ∥_{F}^{2}) .

Then we have $∥ \nabla ψ (Q, R) ∥_{F} \leq 2 γ ω$ , where $ω = {sup}_{θ, θ^{'} \in Θ} ∥ θ - θ^{'} ∥_{F}$ . Also, by (16) we have

F \leq max {∥ Σ_{K_{E}} ∥_{F}, ∥ K_{E} Σ_{K_{E}} K_{E}^{⊤} ∥_{F}} + 2 γ ω .

(24)

Let $max {β_{Q}, β_{R}} / α \leq ι$ for some constant $ι$ . By (15) we have

M \leq ι α \cdot (Tr (Σ_{K_{0}}) + % Tr (K_{0} Σ_{K_{0}} {K_{0}}^{⊤})) .

(25)

By (25) and Lemma 4.1, we obatin

	$∥ Σ_{K_{i}} ∥$	$\leq (α F + 2 M) / α_{Q} \leq F + 2 ι \cdot (Tr (Σ_{K_{0}}) + Tr (K_{0} Σ_{K_{0}} {K_{0}}^{⊤})),$		(26)
	$∥ K_{i} ∥^{2}$			(27)

for all $i \geq 0$ . In §5.2 we further prove that $ν_{V}$ is determined by the uniform upper bound of $∥ Σ_{K_{i}} ∥$ and $∥ K_{i} ∥$ along the solution path, which is established in Lemma 4.1. Hence, by (26) and (27) we have that $ν_{V}$ is independent of $α$ . Meanwhile, by (24) and (25) we have

14 σ_{θ} M (α F + 2 M) = O (α^{3}) .

Thus, for a sufficiently large $α$ , we have

α_{Q} α_{R} α^{2} γ \geq α^{4} γ \geq 14 σ_{θ} ν_{V} M (α F + 2 M),

which leads to Condition 4.2.

Condition 4.2 plays a key role in establishing the convergence. On the one hand, to ensure the boundedness of the cost function $C (K_{i}; θ_{i})$ , we require an upper bound of $λ / η$ in Condition 4.1. On the other hand, to ensure the convergence of the gradient algorithm, we require an upper bound of $η / λ$ in Condtion 4.2. Condition 4.2 ensures such two requirements on stepsizes are compatible.

4.3 Q-Linear Convergence

In this section, we establish the Q-linear convergence of the gradient algorithm in (10) and (11). Recall that the optimal policy takes the form $K^{*} = (B^{⊤} P B + R)^{- 1} B^{⊤} P A$ , where $P$ is the positive definite solution to the discrete-time algebraic Riccati equation (Anderson and Moore, 2007),

f (P, Q, R) = P - A^{⊤} P A - Q + A^{⊤} P B (B^{⊤} P B + R)^{- 1} B^{⊤} P A = 0.

(28)

We denote by $P^{*} (Q, R)$ the corresponding implicit matrix-valued function defined by (28). Also, we define $Y \in R^{d^{2} \times d^{2}}$ as

Y_{(i - 1) d + j, (k - 1) d + ℓ} = \frac{\partial f_{i, j}}{\partial P_{k, ℓ}} (P^{*} (Q^{*}, R^{*}), Q^{*}, R^{*}),

for $i, j, k, ℓ \in [d]$ . We assume the following regularity condition on $f$ .

The unique stationary point of cost parameter $(Q^{*}, R^{*})$ is an interior point of $Θ$ . Also, we assume that $det (Y) \neq 0$ .

We define $K^{*} (θ)$ as the unique optimal policy corresponding to the cost parameter $θ$ and denote by $m^{*} (θ)$ the corresponding value of the objective function $m (K, θ)$ defined in (6), that is,

K∗(θ)=\argminK∈KC(K;θ),m∗(θ)=m(K∗(θ),θ).

(29)

The following two lemmas characterize the local properties of the functions $K^{*} (θ)$ and $m^{*} (θ)$ in a neighborhood of the saddle point $(K^{*}, θ^{*})$ of $m (K, θ)$ .

Under Condition 4.3, there exist constants $τ_{K^{*}}$ and $ν_{K^{*}}$ , and a neighborhood $B_{K}$ of $θ^{*}$ , such that $K^{*} (θ)$ is $τ_{K^{*}}$ -Lipschitz continuous and $ν_{K^{*}}$ -smooth with respect to $θ \in B_{K}$ .

Proof.

See §D.4 for a detailed proof. ∎

Under Condition 4.3, there exist a constant $ν_{m^{*}}$ and a neighborhood $B_{m^{*}}$ of $θ^{*}$ such that $m^{*} (θ)$ is $γ$ -strongly concave and $ν_{m^{*}}$ -smooth with respect to $θ \in B_{m^{*}}$ .

Proof.

See §D.5 for a detailed proof. ∎

To establish the Q-linear convergence, we need an additional condition, which upper bounds the stepsizes $η$ and $λ$ . For the stepsizes $η$ and $λ$ in (10) and (11), let

η \leq 2 / (α_{R} μ + ν_{K^{*}}), λ \leq 2 / (γ + ν_{m^{*}}) .

We define the following potential function

Z_{i} = ∥ θ_{i} - θ^{*} ∥_{F} + a \cdot ∥ K_{i} - K^{*} (θ_{i}) ∥_{F},

(30)

where $a = γ / (3 τ_{K^{*}} ν_{m^{*}})$ . Note that ${lim}_{i \to \infty} Z_{i} = 0$ implies that ${(K_{i}, θ_{i})}_{i = 0}^{\infty}$ converges to $(K^{*}, θ^{*})$ , since we have $K^{*} (θ^{*}) = K^{*}$ . Also, we define

υ = max {1 - λ γ + a \cdot λ ν_{m^{*}} τ_{K^{*}}, (1 - η α_{R} μ) \cdot (1 + λ τ_{V} / a + λ τ_{V} τ_{K^{*}})} .

(31)

The following theorem establishes the Q-linear convergence of the gradient algorithm.

Under Conditions 4.1, 4.2, 4.2, 4.3, and 4.3, we have $υ \in (0, 1)$ in (31). There exists an iteration index $N > 0$ such that $Z_{i + 1} \leq υ \cdot Z_{i}$ for all $i > N$ .

Proof.

See §5.3 for a detailed proof. ∎

5 Proof Sketch

In this section, we sketch the proof of the main results in §4.

5.1 Proof of Stability Guarantee

To prove Lemma 4.1, we lay out two auxiliary lemmas that characterize the geometry of the cost function $C (K; θ)$ with respect to $K$ . The first lemma characterizes the stationary point of policy optimization. The second lemma shows that $C (K; θ)$ is gradient dominated with respect to $K$ .

If $\nabla_{K} C (K; θ) = 0$ , then $K$ is the unique optimal policy corresponding to the cost parameter $θ$ .

Proof.

See §D.2 for a detailed proof. ∎

[Corollary 5 in Fazel et al. (2018)] The cost function $C (K; θ)$ is gradient dominated with respect to $K$ , that is,

C (K; θ) - C (K^{*} (θ); θ) \leq μ_{C} \cdot ∥ \nabla_{K} C (K; θ) ∥_{F}^{2},

where $μ_{C} = ∥ Σ_{K^{*} (θ)} ∥ / (μ^{2} σ_{min} (R))$ and $K^{*} (θ)$ is defined in (29).

Proof.

See Fazel et al. (2018) for a detailed proof. ∎

Lemma 5.1 allows us to upper bound the increment of the cost function at each iteration in (10) and (11) by choosing a sufficiently small $λ$ relative to $η$ . In fact, we construct a threshold such that when $C (K_{i}; θ_{i})$ is close to such a threshold, an upper bound of the increment $C (K_{i + 1}; θ_{i + 1}) - C (K_{i}; θ_{i})$ goes to zero. Thus, $C (K_{i}; θ_{i})$ is upper bounded by such a threshold. See §A for a detailed proof.

5.2 Proof of Global Convergence

To prove Theorem 4.2, we first establish the Lipschitz continuity and smoothness of $m (K, θ)$ in $K$ within a restricted domain $K^{†}$ as in Lemma 4.2. Recall that $V (K)$ is defined in (20) and $m (K, θ)$ takes the form in (21). Since the matrix-valued function $Σ_{K}$ plays a key role in $V (K)$ , in the sequel we characterize the smoothness of $Σ_{K}$ with respect to $K$ within a restricted set. For any constant $S > 0$ , there exist constants $τ_{Σ}$ and $ν_{Σ}$ depending on $S$ such that

∥ Σ_{K} - Σ_{K^{'}} ∥_{F} \leq τ_{Σ} \cdot ∥ K - K^{'} ∥_{F}, ∥ \nabla_{K} (Σ_{K})_{j, ℓ} - \nabla_{K} (Σ_{K^{'}})_{j, ℓ} ∥_{F} \leq ν_{Σ} / d \cdot ∥ K - K^{'} ∥_{F}

for any $K, K^{'} \in {K \in R^{k \times d} | ∥ Σ_{K} ∥ \leq S}$ and $j, ℓ \in [d]$ .

Proof.

See §D.3 for a detailed proof. ∎

Based on Lemmas 5.2 and 4.1, we now prove Lemma 4.2.

Proof.

Let the set $K^{†}$ in Lemma 4.2 be

K^{†} = {K \in R^{k \times d} | ∥ Σ_{K} ∥ \leq (α F + 2 M) / α_{Q}} .

Then by Lemma 4.1, we have

∥ Σ_{K_{i}} ∥ \leq C (K_{i}; θ_{i}) / α_{Q} \leq α (F + 2 M) / α_{Q}

for all $i \geq 0$ , which implies $K_{i} \in K^{†}$ for all $i \geq 0$ . By Lemma 5.2, we obtain the Lipschitz continuity and smoothness of $Σ_{K}$ over $K^{†}$ . Furthermore, by the definition of $V (K)$ in (20) and the boundedness of $K_{i}$ established in Lemma 4.1, $V (K)$ is also Lipschitz continuous and smooth over $K^{†}$ . Thus, we conclude the proof of Lemma 4.2. ∎

Based on Lemma 4.2, we prove the global convergence in Theorem 4.2. To this end, we construct a potential function that decays monotonically along the solution path, which takes the form

P_{i} = m (K_{i}, θ_{i}) + s \cdot ((1 + η ν_{V} σ_{θ}) / 2 \cdot ∥ K_{i + 1} - K_{i} ∥_{F}^{2} + (η / λ - η γ + η λ ν^{2}) / 2 \cdot ∥ θ_{i + 1} - θ_{i} ∥_{F}^{2})

(32)

for some constant $s > 0$ , which is specified in the next lemma. Meanwhile, we define three constants $ϕ_{1}$ , $ϕ_{2}$ , and $ϕ_{3}$ as

$ϕ_{1}$	$= 1 / (2 η) - τ_{V} / 2 - s \cdot (η λ τ_{V}^{2} + 3 η ν_{V} σ_{θ}),$	(33)
$ϕ_{2}$	$= s \cdot (η γ - η λ ν^{2}) / 2 - (1 / λ + τ_{V} + ν) / 2,$	(34)
$ϕ_{3}$	$= s \cdot (η γ - η λ ν^{2}) / 2 - (1 / λ + ν) / 2.$	(35)

The following lemma characterizes the decrement of the potential function defined in (32) at each iteration.

Under Conditions 4.1, 4.2, and 4.2, we have

P_{i + 1} - P_{i} \leq - ϕ_{1} \cdot ∥ K_{i + 1} - K_{i} ∥_{F}^{2} - ϕ_{2} \cdot ∥ θ_{i + 1} - θ_{i} ∥_{F}^{2} - ϕ_{3} \cdot ∥ θ_{i} - θ_{i - 1} ∥_{F}^{2} .

(36)

Moreover, we have $ϕ_{1}, ϕ_{2}, ϕ_{3} > 0$ for $s = 12 / (13 η^{2} ν_{V} σ_{θ})$ .

Proof.

See §B for a detailed proof. ∎

Based on Lemma 5.2, we now prove Theorem 4.2.

Proof.

By the definitions of $P_{i}$ and $m (K, θ)$ in (32) and (6), we have $P_{i} \geq P - -$ for all $i \geq 0$ , where $P - -$ is

P - - = inf θ \in Θ {- C (K_{E}; θ) - ψ (θ)} > - \infty .

Here we use the fact $C (K; θ) \geq 0$ for any $K \in K$ and $θ \in Θ$ . Let $ϕ = 1 / min {ϕ_{1}, ϕ_{2}}$ , where $ϕ_{1}$ and $ϕ_{2}$ are defined in (33) and (34). By rearranging the terms in (36), we obtain

\infty \sum i = 0 ∥ K_{i + 1} - K_{i} ∥_{F}^{2} + ∥ θ_{i + 1} - θ_{i} ∥_{F}^{2} \leq \infty \sum i = 0 ϕ \cdot (P_{i} - P_{i + 1}) \leq ϕ \cdot (P_{0} - P - -) < \infty .

By (10) and (11), we have

\infty \sum i = 0 ∥ L (K_{i}, θ_{i}) ∥_{F}^{2} \leq \infty \sum i = 0 ϕ^{'} \cdot (∥ K_{i + 1} - K_{i} ∥_{F}^{2} + ∥ θ_{i + 1} - θ_{i} ∥_{F}^{2}) \leq ϕ^{'} ϕ \cdot (P_{0} - P - -) < \infty,

where $ϕ^{'} = max {1, 1 / η^{2}, 1 / λ^{2}}$ , which implies that ${∥ L (K_{i}, θ_{i}) ∥_{F}}_{i = 0}^{\infty}$ converges to zero. Also, let

ζ = ϕ^{'} ϕ \cdot (P_{0} - P - -) .

(37)

For any $ε > 0$ , by the definition of $Γ (ε)$ in (23), we have

ε \cdot Γ (ε) \leq Γ (ε) \sum i = 0 ∥ L (K_{i}, θ_{i}) ∥_{F}^{2} \leq ϕ^{'} ϕ \cdot (P_{0} - P - -) = ζ,

which implies

On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator

Authors

Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate

When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence

On Computation and Generalization of Generative Adversarial Imitation Learning

Generative Adversarial Self-Imitation Learning

Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration

Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Minimax Problems

Fail-Safe Generative Adversarial Imitation Learning

1 Introduction

2 Background

2.1 Linear Quadratic Regulator

2.2 Imitation Learning

3 Algorithm

3.1 Minimax Formulation

3.2 Gradient Algorithm

4 Main Results

4.1 Stability Guarantee

Proof.

4.2 Global Convergence

Proof.

Proof.

Proof.

4.3 Q-Linear Convergence

Proof.

Proof.

Proof.

5 Proof Sketch

5.1 Proof of Stability Guarantee

Proof.

Proof.

5.2 Proof of Global Convergence

Proof.

Proof.

Proof.

Proof.

On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator

Authors

Related Research

Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate

When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence

On Computation and Generalization of Generative Adversarial Imitation Learning

Generative Adversarial Self-Imitation Learning

Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration

Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Minimax Problems

Fail-Safe Generative Adversarial Imitation Learning

This week in AI

1 Introduction

2 Background

2.1 Linear Quadratic Regulator

2.2 Imitation Learning

3 Algorithm

3.1 Minimax Formulation

3.2 Gradient Algorithm

4 Main Results

4.1 Stability Guarantee

Proof.

4.2 Global Convergence

Proof.

Proof.

Proof.

4.3 Q-Linear Convergence

Proof.

Proof.

Proof.

5 Proof Sketch

5.1 Proof of Stability Guarantee

Proof.

Proof.

5.2 Proof of Global Convergence

Proof.

Proof.

Proof.

Proof.