Super-efficiency of automatic differentiation for functions defined as a minimum

1 Introduction

In machine learning, many objective functions are expressed as the minimum of another function: functions

ℓ

defined as

ℓ (x) = min z \in R^{m} L (z, x),

(1)

where $L : R^{m} \times R^{n} \to R$ . Such formulation arises for instance in dictionary learning, where $x$ is a dictionary, $z$ a sparse code, and $L$ is the Lasso cost (Mairal et al., 2010). In this case, $ℓ$ measures the ability of the dictionary $x$ to encode the input data. Another example is the computation of the Wassertein barycenter of distributions in optimal transport (Agueh and Carlier, 2011): $x$ represents the barycenter, $ℓ$ is the sum of distances to $x$ , and the distances themselves are defined by minimizing the transport cost. In the field of optimization, formulation (1) is also encountered as a smoothing technique, for instance in reweighted least-squares (Daubechies et al., 2010) where $L$ is smooth but not $ℓ$

. In game theory, such problems naturally appear in two-players maximin games

(von Neumann, 1928), with applications for instance to generative adversarial nets (Goodfellow et al., 2014). In this setting,

ℓ

should be maximized.

A key point to optimize $ℓ$ – either maximize or minimize – is usually to compute the gradient of $ℓ$ , $g^{*} (x) ≜ \nabla_{x} ℓ (x)$ . If the minimizer $z^{*} (x) = {a r g m i n}_{z} L (z, x)$ is available, the first order optimality conditions impose that $\nabla_{z} L (z^{*} (x), x) = 0$ and the gradient is given by

g^{*} (x) = \nabla_{x} L (z^{*} (x), x) .

(2)

However, in most cases the minimizer $z^{*} (x)$ of the function is not available in closed-form. It is approximated via an iterative algorithm, which produces a sequence of iterates $z_{t} (x)$ . There are then three ways to estimate $g^{*} (x)$ :

The analytic estimator corresponds to plugging the approximation $z_{t} (x)$ in (2)

g_{t}^{1} (x) ≜ \nabla_{x} L (z_{t} (x), x) .

(3)

The automatic estimator is $g_{t}^{2} (x) ≜ \partial_{x} [L (z_{t} (x), x)]$ , where the derivative is computed with respect to $z_{t} (x)$

as well. The chain rule gives

g_{t}^{2} (x) = \nabla_{x} L (z_{t} (x), x) + \frac{\partial z_{t}}{\partial x} \nabla_{z} L (z_{t} (x), x) .

(4)

This expression can be computed efficiently using automatic differentiation (Baydin et al., 2018), in most cases at a cost similar to that of computing $z_{t} (x)$ .

If $\nabla_{z z} L (z^{*} (x), x)$ is invertible, the implicit function theorem gives $\frac{\partial z^{*} (x)}{\partial x} = J (z^{*} (x), x)$ where $J (z, x) ≜ - \nabla_{x z} L (z, x) {[\nabla_{z z} L (z, x)]}_{z z}^{- 1}$ . The implicit estimator is

g_{t}^{3} (x) ≜ \nabla_{x} L (z_{t} (x), x) + J (z_{t} (x), x) \nabla_{z} L (z_{t} (x), x) .

(5)

This estimator can be more costly to compute than the previous ones, as a $m \times m$ linear system has to be solved.

These estimates have been proposed and used by different communities. The analytic one corresponds to alternate optimization of $L$ , where one updates $x$ while considering that $z$ is fixed. It is used for instance in dictionary learning (Olshausen and Field, 1997; Mairal et al., 2010) or in optimal transport (Feydy et al., 2019)

. The second is common in the deep learning community as a way to differentiate through optimization problems

(Gregor and Le Cun, 2010). Recently, it has been used as a way to accelerate convolutional dictionary learning (Tolooshams et al., 2018). It has also been used to differentiate through the Sinkhorn algorithm in optimal transport applications (Boursier and Perchet, 2019; Genevay et al., 2018). It integrates smoothly in a machine learning framework, with dedicated libraries (Abadi et al., 2016; Paszke et al., 2019)

. The third one is found in bi-level optimization, for instance for hyperparameter optimization

(Bengio, 2000)

. It is also the cornerstone of the use of convex optimization as layers in neural networks

(Agrawal et al., 2019).

Contribution In this article, we want to answer the following question: which one of these estimators is the best? The central result, presented in section 2, is the following convergence speed, when $L$ is differentiable and under mild regularity hypothesis (Proposition 1, 2 and 3)

	$\| g_{t}^{1} (x) - g^{*} (x) \|$	$= O (\| z_{t} (x) - z^{*} (x) \|),$
	$\| g_{t}^{2} (x) - g^{*} (x) \|$	$= o (\| z_{t} (x) - z^{*} (x) \|),$
	$\| g_{t}^{3} (x) - g^{*} (x) \|$	$= O (\| z_{t} (x) - z^{*} (x) \|^{2}) .$

This is a super-efficiency phenomenon for the automatic estimator, illustrated in Figure 1 on a toy example. As our analysis reveals, the bound on $g^{2}$ depends on the convergence speed of the Jacobian of $z_{t}$ , which itself depends on the optimization algorithm used to produce $z_{t}$ . In section 3, we build on the work of Gilbert (1992) and give accurate bounds on the convergence of the Jacobian for gradient descent (Proposition 5) and stochastic gradient descent (Proposition 8 and 9) in the strongly convex case. We then study a simple case of non-strongly convex problem (Proposition 12). To the best of our knowledge, these bounds are novel. This analysis allows us to refine the convergence rates of the gradient estimators.

Figure 1: Convergence of the gradient estimators. $L$ is strongly convex, $x$ is a random point and $z_{t} (x)$ corresponds to $t$ iterations of gradient descent. As $t$ increases, $z_{t}$ goes to $z^{*}$ at a linear rate. $g_{t}^{1}$ converges at the same rate while $g_{t}^{2}$ and $g_{t}^{3}$ are twice as fast.

In section 4, we start by recalling and extending the consequence of using wrong gradients in an optimization algorithm (Proposition 13 and 14). Then, since each gradient estimator comes at a different cost, we put the convergence bounds developed in the paper in perspective with a complexity analysis. This leads to practical and principled guidelines about which estimator should be used in which case. Finally, we provide numerical illustrations of the aforementioned results in section 5.
Notation The $ℓ^{2}$ norm of $z \in R^{m}$ is $| z | = \sqrt{\sum_{i = 1}^{m} z_{i}^{2}}$ . The operator norm of $M \in R^{m \times n}$ is $∥ M ∥ = {sup}_{| z | = 1} | M z |$ and the Frobenius norm is $∥ M ∥_{F} = \sqrt{\sum_{i, j} M_{i j}^{2}}$

. The vector of size

n

full of

1^{'}

s is

1_{n}

. The Euclidean scalar product is

⟨ \cdot, \cdot ⟩

The proofs are only sketched in the article, full proofs are deferred to appendix.

2 Convergence speed of gradient estimates

We consider a compact set $K = K_{z} \times K_{x} \subset R^{m} \times R^{n}$ . We make the following assumptions on $L$ .
H1: $L$ is twice differentiable over $K$ with second derivatives $\nabla_{x z} L$ and $\nabla_{z z} L$ respectively $L_{x z}$ and $L_{z z}$ -Lipschitz.
H2:For all $x \in K_{x}$ , $z \to L (z, x)$ has a unique minimizer $z^{*} (x) \in int (K_{z})$ . The mapping $z^{*} (x)$ is differentiable, with Jacobian $J^{*} (x) \in R^{n \times m}$ .
H1 implies that $\nabla_{z} L$ and $\nabla_{x} L$ are Lipschitz, with constants $L_{z}$ and $L_{x}$ . The Jacobian of $z_{t}$ at $x \in K_{x}$ is $J_{t} ≜ \frac{\partial z_{t} (x)}{\partial x} \in R^{n \times m}$ . For the rest of the section, we consider a point $x \in K_{x}$ , and we denote $g^{*} = g^{*} (x)$ , $z^{*} = z (x)$ and $z_{t} = z_{t} (x)$ .

2.1 Analytic estimator $g^{1}$

The analytic estimator (3) approximates $g^{*}$ as well as $z_{t}$ approximates $z^{*}$ by definition of the $L_{x}$ -smoothness.

Proposition 1 (Convergence of the analytic estimator).

The analytic estimator verifies $| g_{t}^{1} - g^{*} | \leq L_{x} | z_{t} - z^{*} |$ .

2.2 Automatic estimator $g^{2}$

The automatic estimator (4) can be written as

g^{2} = g^{*} + R (J_{t}) (z_{t} - z^{*}) + R_{x z} + J_{t} R_{z z}

(6)

where

	$R_{x z}$	$≜ \nabla_{x} L (z_{t}, x) - \nabla_{x} L (z^{}, x) - \nabla_{x z} L (z^{}, x) (z_{t} - z^{*})$
	$R_{z z}$	$≜ \nabla_{z} L (z_{t}, x) - \nabla_{z z} L (z^{}, x) (z_{t} - z^{}) .$

are Taylor’s rests and

R (J) ≜ J \nabla_{z z} L (z^{*}, x) + \nabla_{x z} L (z^{*}, x) .

(7)

The implicit function theorem states that $R (J^{*}) = 0$ . Importantly, in a non strongly-convex setting where $\nabla_{z z} L (z^{*}, x)$ is not invertible, it might happen that $R (J_{t})$ goes to $0$ even though $J_{t}$ does not converge to $J^{*}$ . H1 implies a quadratic bound on the rests

| R_{x z} | \leq \frac{L_{x z}}{2} | z_{t} - z^{*} |^{2} and | R_{z z} | \leq \frac{L_{z z}}{2} | z_{t} - z^{*} |^{2} .

(8)

We assume that $J_{t}$ is bounded $∥ J_{t} ∥ \leq L_{J}$ . This holds when $J_{t}$ converges, which is the subject of section 3. The triangle inequality in Equation 6 gives:

Proposition 2 (Convergence of the automatic estimator).

We define

L ≜ L_{x z} + L_{J} L_{z z}

(9)

Then $| g_{t}^{2} - g^{*} | \leq ∥ R (J_{t}) ∥ | z_{t} - z^{*} | + \frac{L}{2} | z_{t} - z^{*} |^{2}$ .

This proposition shows that the rate of convergence of $g^{2}$ depends on the speed of convergence of $R (J_{t})$ . For instance, if $R (J_{t})$ goes to $0$ , we have

g_{t}^{2} - g^{*} = o (| z_{t} - z^{*} |) .

Unfortunately, it might happen that, even though $z_{t}$ goes to $z^{*}$ , $R (J_{t})$ does not go to $0$ since differentiation is not a continuous operation. In section 3, we refine this convergence rate by analyzing the convergence speed of the Jacobian in different settings.

2.3 Implicit estimator $g^{3}$

The implicit estimator (5) is well defined provided that $\nabla_{z z} L$ is invertible. We obtain convergence bounds by making a Lipschitz assumption on $J (z, x) = - \nabla_{x z} L (z, x) {[\nabla_{z z} L (z, x)]}_{z z}^{- 1}$ .

Proposition 3.

[Convergence of the implicit estimator] Assume that $J$ is $L_{J}$ -Lipschitz with respect to its first argument, and that $∥ J_{t} ∥ \leq L_{J}$ . Then, for $L$ as defined in (9),

| g_{t}^{3} - g^{*} | \leq (\frac{L}{2} + L_{J} L_{z}) | z_{t} - z^{*} |^{2} .

(10)

Sketch of proof.

The proof is similar to that of Proposition 2, using $∥ R (J (z_{t}, x)) ∥ \leq L_{z} L_{J} | z_{t} - z^{*} |$ . ∎

Therefore this estimator converges twice as fast as $g^{1}$ , and at least as fast as $g^{2}$ . Just like $g^{1}$ this estimator does not need to store the past iterates in memory, since it is a function of $z_{t}$ and $x$ . However, it is usually much more costly to compute.

2.4 Link with bi-level optimization

Bi-level optimization appears in a variety of machine-learning problems, such as hyperparameter optimization (Pedregosa, 2016) or supervised dictionary learning (Mairal et al., 2012). It considers problems of the form

min x \in R^{n} ℓ^{'} (x) ≜ L^{'} (z^{*} (x), x) s.t. z^{*} (x) \in a r g m i n z \in R^{m} L (z, x),

(11)

where $L^{'} : R^{m} \times R^{n} \to R$ is another objective function. The setting of our paper is a special instance of bi-level optimization where $L^{'} = L$ . The gradient of $ℓ^{'}$ is

g^{' *} = \nabla_{x} ℓ^{'} (x) = \nabla_{x} L^{'} (z^{*}, x) + J^{*} \nabla_{z} L^{'} (z^{*}, x) .

When $z_{t} (x)$ is a sequence of approximate minimizers of $L$ , gradient estimates can be defined as

	$g^{' 1}$	$= \nabla_{x} L^{'} (z_{t} (x), x),$
	$g^{' 2}$	$= \nabla_{x} L^{'} (z_{t} (x), x) + J_{t} \nabla_{z} L^{'} (z_{t} (x), x),$
	$g^{' 3}$	$= \nabla_{x} L^{'} (z_{t} (x), x) + J (z_{t} (x), x) \nabla_{z} L^{'} (z_{t} (x), x) .$

Here, $\nabla_{z} L^{'} (z^{*}, x) \neq 0$ , since $z^{*}$ does not minimize $L^{'}$ . Hence, $g^{' 1}$ does not estimate $g^{' *}$ . Moreover, in general,

\nabla_{x z} L^{'} (z^{*}, x) + J^{*} \nabla_{z z} L^{'} (z^{*}, x) \neq 0.

Therefore, there is no cancellation to allow super-efficiency of $g^{2}$ and $g^{3}$ and we only obtain linear rates

g^{' 2} - g^{*} = O (| z_{t} - z^{*} |), g^{' 3} - g^{*} = O (| z_{t} - z^{*} |) .

3 Convergence speed of the Jacobian

In order to get a better understanding of the convergence properties of the gradient estimators – in particular $g^{2}$ – we analyze it in different settings. A large portion of the analysis is devoted to the convergence of $R (J_{t})$ to $0$ , since it does not directly follow from the convergence of $z_{t}$ . In most cases, we show convergence of $J_{t}$ to $J^{*}$ , and use

∥ R (J_{t}) ∥ \leq L_{z} ∥ J_{t} - J^{*} ∥

(12)

in the bound of Proposition 2.

3.1 Contractive setting

When $z_{t}$ are the iterates of a fixed point iteration with a contractive mapping, we recall the following result due to Gilbert (1992).

Proposition 4 (Convergence speed of the Jacobian).

Assume that $z_{t}$ is produced by a fixed point iteration

z_{t + 1} = Φ (z_{t}, x),

where $Φ : K_{z} \times K_{x} \to K_{z}$ is differentiable. We suppose that $Φ$ is contractive: there exists $κ < 1$ such that for all $(z, z^{'}, x) \in K_{z} \times K_{z} \times K_{x}$ , $| Φ (z, x) - Φ (z^{'}, x) | \leq κ | z - z^{'} | .$ Under mild regularity conditions on $Φ$ :

$z_{t}$ converges to a differentiable function $z^{*}$ such that $z^{*} = Φ (z^{*}, x)$ , with Jacobian $J^{*}$ .
$| z_{t} - z^{*} | = O (κ^{t})$ and $∥ J_{t} - J^{*} ∥ = O (t κ^{t})$

3.2 Gradient descent in the strongly convex case

We consider the gradient descent iterations produced by the mapping $Φ (z, x) = z - ρ \nabla_{z} L (z, x)$ , with a step-size $ρ \leq 1 / L_{z}$ . We assume that $L$ is $μ$ -strongly convex with respect to $z$ , i.e. $\nabla_{z z} L ⪰ μ Id$ for all $z \in K_{z}, x \in K_{x}$ . In this setting, $Φ$ satisfies the hypothesis of Proposition 4, and we obtain precise bounds.

Proposition 5.

[Convergence speed of the Jacobian of gradient descent in a strongly convex setting] Let $z_{t}$ produced by the recursion $z_{t + 1} = z_{t} - ρ \nabla_{z} L (z_{t}, x)$ with $ρ \leq 1 / L_{z}$ and $κ ≜ 1 - ρ μ$ . We have $| z_{t} - z^{*} | \leq κ^{t} | z_{0} - z^{*} |$ and $∥ J_{t} - J^{*} ∥ \leq t κ^{t - 1} ρ L | z_{0} - z^{*} |$ where $L$ is defined in (9).

Sketch of proof (c.1).

We show that $δ_{t} = ∥ J_{t} - J^{*} ∥$ satisfies the recursive inequality $δ_{t + 1} \leq κ δ_{t} + ρ L | z_{0} - z^{*} | κ^{t}$ . ∎

As a consequence, Prop. 1, 2, 3 together with Eq. (12) give in this case

$\| g^{1} - g^{*} \|$	$\leq L_{x} \| z_{0} - z^{*} \| κ^{t},$
$\| g^{2} - g^{*} \|$	$\leq (ρ L_{z} t + \frac{κ}{2}) L \| z_{0} - z^{*} \|^{2} κ^{2 t - 1},$	(13)
$\| g^{3} - g^{*} \|$	$\leq (\frac{L}{2} + L_{J} L_{z}) \| z_{0} - z^{*} \|^{2} κ^{2 t} .$

We get the convergence speed $g^{2} - g^{*} = O (t κ^{2 t})$ , which is almost twice better than the rate for $g^{1}$ . Importantly, the order of magnitude in Proposition 5 is tight, as it can be seen in Proposition 15.

3.3 Stochastic gradient descent in $z$

We provide an analysis of the convergence of $J_{t}$ in the stochastic gradient descent setting, assuming once again the $μ$ -strong convexity of $L$ . We suppose that $L$ is an expectation

L (z, x) = E_{ξ} [C (z, x, ξ)],

where $ξ$ is drawn from a distribution $d$ , and $C$ is twice differentiable. Stochastic gradient descent (SGD) with steps $ρ_{t}$ iterates

z_{t + 1} (x) = z_{t} (x) - ρ_{t} \nabla_{z} C (z_{t} (x), x, ξ_{t + 1}) % where ξ_{t + 1} \sim d .

In the stochastic setting, Proposition 2 becomes

Proposition 6.

Define

δ_{t} = E [∥ J_{t} - J^{*} ∥_{F}^{2}] and d_{t} = E [| z_{t} - z^{*} |^{2}] .

(14)

We have $E [| g^{2} - g^{*} |] \leq L_{z} \sqrt{δ_{t}} \sqrt{d_{t}} + \frac{L}{2} d_{t}$ .

Sketch of proof (c.3).

We use Cauchy-Schwarz and the norm inequality $∥ \cdot ∥ \leq ∥ \cdot ∥_{F}$ to bound $E [∥ R (J_{t}) ∥ | z_{t} - z^{*} |]$ . ∎

We begin by deriving a recursive inequality on $δ_{t}$ , inspired by the analysis techniques of $d_{t}$ .

Proposition 7.

[Bounding inequality for the Jacobian] We assume bounded Hessian noise, in the sense that $E [∥ \nabla_{z z} C (z, x, ξ) ∥_{F}^{2}] \leq σ_{z z}^{2}$ and $E [∥ \nabla_{x z} C (z, x, ξ) ∥_{F}^{2}] \leq σ_{x z}^{2}$ . Let $r = min (n, m)$ , and $B^{2} = σ_{x z}^{2} + L_{J}^{2} σ_{z z}^{2}$ . We have

δ_{t + 1} \leq (1 - 2 ρ_{t} μ) δ_{t} + 2 ρ_{t} \sqrt{r} L \sqrt{d_{t}} \sqrt{} δ_{t} + ρ_{t}^{2} B^{2} .

(15)

Sketch of proof (c.4).

A standard strong convexity argument gives the bound

δ_{t + 1} \leq (1 - 2 ρ_{t} μ) δ_{t} + 2 ρ_{t} \sqrt{r} L E [∥ J_{t} - J^{*} ∥_{F} | z_{t} - z_{0} |] + ρ_{t}^{2} B^{2} .

The middle term is then bounded using Cauchy-Schwarz inequality. ∎

Therefore, any convergence bound on $d_{t}$ provides another convergence bound on $δ_{t}$ by unrolling Eq. (15). We first analyze the fixed step-size case by using the simple “bounded gradients” hypothesis and bounds on $d_{t}$ from Moulines and Bach (2011)

. In this setting, the iterates converge linearly until they reach a threshold caused by gradient variance.

Proposition 8.

[SGD with constant step-size] Assume that the gradients have bounded variance $E_{ξ} [| \nabla_{z} C (z, x, ξ) |^{2}] \leq σ^{2}$ . Assume $ρ_{t} = ρ < 1 / L_{z}$ , and let $κ_{2} = \sqrt{1 - 2 ρ μ}$ and $β = \sqrt{\frac{σ^{2} ρ}{2 μ}}$ . In this setting

δ_{t} \leq {(κ_{2}^{t} (∥ J^{*} ∥_{F} + t α) + B_{2})}_{2}^{2},

where $α = \frac{ρ \sqrt{r} L}{κ_{2}} | z^{*} - z_{0} |$ and $B_{2} = \frac{ρ \sqrt{r} L β}{κ_{2} (1 - κ_{2})} + \frac{ρ B}{(1 - κ_{2})}$ .

Sketch of proof (c.5).

Moulines and Bach (2011) give $d_{t} \leq κ_{2}^{2 t} | z_{0} - z^{*} |^{2} + β^{2}$ , which implies $\sqrt{d_{t}} \leq κ_{2}^{t} | z_{0} - z^{*} | + β .$ A bit of work on Eq. (15) then gives

\sqrt{δ_{t + 1}} \leq κ_{2} \sqrt{δ_{t}} + ρ κ_{2}^{t} + (1 - κ_{2}) B_{2}

Unrolling the recursion gives the proposed bound. ∎

This bound showcases that the familiar “two-stages” behavior also stands for $δ_{t}$ : a transient linear decay at rate $(1 - 2 ρ μ)^{t}$ in the first iterations, and then convergence to a stationary noise level $B_{2}^{2}$ . This bound is not tight enough to provide a good estimate of the noise level in $δ_{t}$ . Let $B_{3}^{2}$ the limit of the sequence defined by the recursion (15). We find

B_{3}^{2} = (1 - 2 ρ μ) B_{3}^{2} + 2 ρ \sqrt{r} L β B_{3} + ρ^{2} B^{2},

which gives by expanding $β$

B_{3} = \sqrt{ρ} \frac{\sqrt{r} L σ}{(2 μ)^{\frac{3}{2}}} ⎛ ⎝ 1 + \sqrt{1 + \frac{4 μ^{2} B^{2}}{r L^{2} σ^{2}}} ⎞ ⎠ .

This noise level scales as $\sqrt{ρ}$ , which is observed in practice. In this scenario, Prop. 1, 2 and 3 show that $g^{1} - g^{*}$ reaches a noise level proportional to $\sqrt{ρ}$ , just like $d_{t}$ , while both $g^{2} - g^{*}$ and $g^{3} - g^{*}$ reach a noise level proportional to $ρ$ : $g^{2}$ and $g^{3}$ can estimate $g^{*}$ to an accuracy that can never be reached by $g^{1}$ . We now turn to the decreasing step-size case.

Proposition 9.

[SGD with decreasing step-size] Assume that $ρ_{t} = ρ_{0} t^{- α}$ with $α \in (0, 1)$ . Assume a bound on $d_{t}$ of the form $d_{t} \leq d^{2} t^{- α}$ . Then

δ_{t} \leq 4 \frac{ρ_{0} B^{2} μ + r L^{2} d^{2}}{μ^{2}} t^{- α} + o (t^{- α}) .

Sketch of proof (c.6).

We use (15) to obtain a recursion

δ_{t + 1} \leq (1 - μ ρ_{0} t^{- α}) δ_{t} + (B^{2} ρ_{0}^{2} + \frac{r L^{2} d^{2} ρ_{0}}{μ}) t^{- 2 α},

which is then unrolled. ∎

When $ρ_{t} \propto t^{- α}$ , we have $d_{t} = O (t^{- α})$ so the assumption $d_{t} \leq d^{2} t^{- α}$ is verified for some $d$ . One could use the precise bounds of (Moulines and Bach, 2011) to obtain non-asymptotic bounds on $δ_{t}$ as well.

Overall, we recover bounds for $δ_{t}$ with the same behavior than the bounds for $d_{t}$ . Pluging them in Proposition 6 gives the asymptotic behaviors for the gradient estimators.

Proposition 10 (Convergence speed of the gradient estimators for SGD with decreasing step).

Assume that $ρ_{t} = C t^{- α}$ with $α \in (0, 1)$ . Then

	$E_{ξ} [\| g^{1} - g^{*} \|]$	$= O (\sqrt{t^{- α}}), E_{ξ} [\| g^{2} - g^{*} \|] = O (t^{- α})$
		$E_{ξ} [\| g^{3} - g^{*} \|] = O (t^{- α})$

The super-efficiency of $g^{2}$ and $g^{3}$ is once again illustrated, as they converge at the same speed as $d_{t}$ .

3.4 Beyond strong convexity

All the previous results rely critically on the strong convexity of $L$ . A function $f$ with minimizer $z^{*}$ is $p$ -Łojasiewicz (Attouch and Bolte, 2009) when $μ (f (z) - f (z^{*}))^{p - 1} \leq ∥ \nabla f (z) ∥^{p}$ for some $μ > 0$ . Any strongly convex function is $2$ -Łojasiewicz: the set of $p$ -Łojasiewicz functions for $p \geq 2$ offers a framework beyond strong-convexity that still provides convergence rates on the iterates. The general study of gradient descent on this class of function is out of scope for this paper. We analyze a simple class of $p$ -Łojasiewicz functions, the least mean $p$ -th problem, where

L (z, x) ≜ \frac{1}{p} n \sum i = 1 (x_{i} - [D z]_{i})^{p}

(16)

for $p$ an even integer and $D$ is overcomplete (rank $(D) = n$ ). In this simple case, $L (\cdot, x)$ is minimized by cancelling $x - D z$ , and $g^{*} = (x - D z^{*})^{p - 1} = 0$ .

In the case of least squares ( $p = 2$ ) we can perfectly describe the behavior of gradient descent, which converges linearly.

Proposition 11.

Let $z_{t}$ the iterates of gradient descent with step $ρ \leq \frac{1}{L_{z}}$ in (16) with $p = 2$ , and $z^{*} \in a r g m i n L (z, x)$ . It holds

g^{1} = D (z_{t} - z^{*}), g^{2} = D (z_{2 t} - z^{*}) and g^{3} = 0.

Proof.

The iterates verify $z_{t} - z^{*} = (I - D^{⊤} D)^{t} (z_{0} - z^{*})$ , and we find $J_{t} \nabla_{z} L (z_{t}, x) = (I_{n} - (I_{n} - D^{⊤} D)^{t}) (x - D z_{t})$ . The result follows. ∎

The automatic estimator therefore goes exactly twice as fast as the analytic one to $g^{*}$ , while the implicit estimator is exact. Then, we analyze the case where $p \geq 4$ in a more restrictive setting.

Proposition 12.

For $p \geq 4$ , we assume $D D^{⊤} = I_{n}$ . Let $α ≜ \frac{p - 1}{p - 2}$ . We have

| g_{t}^{1} | = O (t^{- α}), | g_{t}^{2} | = O (t^{- 2 α}), g_{t}^{3} = 0 .

Sketch of proof (c.7).

We first show that the residuals $r_{t} = x - D z_{t}$ verify $r_{t} = {(\frac{1}{ρ (p - 2) t})}^{\frac{1}{p - 2}} (1 + O (\frac{log (t)}{t}))$ , which gives the result for $g^{1}$ . We find $g_{t}^{2} = M_{t} r_{t}^{p - 1}$ where $M_{t} = I_{n} - J_{t} D^{⊤}$ verifies $M_{t + 1} = M_{t} (I_{n} - (p - 1) ρ diag (r_{t}^{p - 2}))$ . Using the development of $r_{t}$ and unrolling the recursion concludes the proof. ∎

For this problem, $g^{2}$ is of the order of magnitude of $g^{1}$ squared and as $p \to + \infty$ , we see that the rate of convergence of $g^{1}$ goes to $t^{- 1}$ , while the one of $g^{2}$ goes to $t^{- 2}$ .

4 Consequence on optimization

In this section, we study the impact of using the previous inexact estimators for first order optimization. These estimators nicely fit in the framework of inexact oracles introduced by Devolder et al. (2014).

4.1 Inexact oracle

We assume that $ℓ$ is $μ_{x}$ -strongly convex and $L_{x}$ -smooth with minimizer $x^{*}$ . A $(δ, μ, L)$ -inexact oracle is a couple $(ℓ_{δ}, g_{δ})$ such that $ℓ_{δ} : R^{m} \to R$ is the inexact value function, $g_{δ} : R^{m} \to R^{m}$ is the inexact gradient and for all $x, y$

\frac{μ}{2} | x - y |^{2} \leq ℓ (x) - ℓ_{δ} (y) - ⟨ g_{δ} (y) | x - y ⟩ \leq \frac{L}{2} | x - y |^{2} + δ .

(17)

Devolder et al. (2013) show that if the gradient approximation $g^{i}$ verifies $| g^{*} (x) - g^{i} (x) | \leq Δ_{i}$ for all $x$ , then $(ℓ, g^{i})$ is a $(δ_{i}, \frac{μ_{x}}{2}, 2 L_{x})$ -inexact oracle, with

δ_{i} = Δ_{i}^{2} (\frac{1}{μ_{x}} + \frac{1}{2 L_{x}}) .

(18)

We consider the optimization of $ℓ$ with inexact gradient descent: starting from $x_{0} \in R^{n}$ , it iterates

x_{q + 1} = x_{q} - η g_{t}^{i} (x_{q}),

(19)

with $η = \frac{1}{2 L_{x}}$ , a fixed $t$ and $i = 1, 2$ or $3$ .

Proposition 13.

[Devolder et al. 2013, Theorem 4] The iterates $x_{q}$ with estimate $g^{i}$ verify

ℓ (x_{q}) - ℓ (x^{*}) \leq 2 L_{x} (1 - \frac{μ_{x}}{4 L_{x}})^{q} | x_{0} - x^{*} |^{2} + δ_{i}

with $δ_{i}$ defined in (18).

As $q$ goes to infinity, the error made on $ℓ (x^{*})$ tends towards $δ_{i} = O (| g_{t}^{i} - g^{*} |^{2})$ . Thus, a more precise gradient estimate achieves lower optimization error. This illustrates the importance of using gradients estimates with an error $Δ_{i}$ as small as possible.

We now consider stochastic optimization for our problem, with loss $ℓ$ defined as

ℓ (x) = E_{υ} [h (x, υ)] with h (x, υ) = min z H (z, x, υ) .

Stochastic gradient descent with constant step-size $η \leq \frac{1}{2 L_{x}}$ and inexact gradients iterates

x_{q + 1} = x_{q} - η g_{t}^{i} (x_{q}, υ_{q + 1}),

where $g_{t}^{i} (x_{q}, υ_{q + 1})$ is computed by an approximate minimization of $z \to H (z, x_{q}, υ_{q + 1})$ .

Proposition 14.

We assume that $H$ is $μ_{x}$ -strongly convex, $L_{x}$ -smooth and verifies

E [| \nabla_{x} h (x, υ) - \nabla_{x} ℓ (x) |^{2}] \leq σ^{2} .

The iterates $x_{q}$ of SGD with approximate gradient $g^{i}$ and step-size $η$ verify

E | x_{q} - x^{*} |^{2} \leq (1 - \frac{η μ_{x}}{2})^{q} | x_{0} - x^{*} | + \frac{2 η}{μ_{x}} σ^{2} + \frac{4}{μ_{x}} δ_{i}

with $δ_{i} = Δ_{i}^{2} (\frac{1}{μ_{x}} + \frac{1}{2 L_{x}} + 2 η)$ .

The proof is deferred to Appendix D. In this case, it is pointless to achieve an estimation error on the gradient $Δ_{i}$ smaller than some fraction of the gradient variance $σ^{2}$ .

As a final note, these results extend without difficulty to the problem of maximizing $ℓ$ , by considering gradient ascent or stochastic gradient ascent.

4.2 Time and memory complexity

In the following, we put our results in perspective with a computational and memory complexity analysis, allowing us to provide practical guidelines for optimization of $ℓ$ .

Gradient estimate	Computational cost
$g_{t}^{1}$	$O (m n t)$
$g_{t}^{2}$	$O (c m n t)$
$g_{t}^{3}$	$O (m n t + m^{3} + m^{2} n))$

Table 1: Computational cost for a quadratic loss

L

. Here

c \geq 1

corresponds to the relative added cost of automatic differentiation.

Computational complexity of the estimators The cost of computing the estimators depends on the cost function $L$ . We give a complexity analysis in the least squares case (16) which is summarized in Table 1. In this case, computing the gradient $\nabla_{z} L$ takes $O (m n)$ operations, therefore the cost of computing $z_{t}$ with gradient descent is $O (m n t)$ . Computing $g_{t}^{1}$ comes at the same price. The estimator $g^{2}$ requires a reverse pass on the computational graph, which costs a factor $c \geq 1$ of the forward computational cost: the final cost is $O (c m n)$ . Griewank and Walther (2008) showed that typically $c \in [2, 3]$ . Finally, computing $g^{3}$ requires a costly $O (m^{2} n)$ Hessian computation, and a $O (m^{3})$ linear system inversion. The final cost is $O (m n t + m^{3} + m^{2} n)$ . The linear scaling of $g_{t}^{1}$ and $g_{t}^{2}$ is highlighted in Figure A.3. In addition, computing $g^{2}$ usually requires to store in memory all intermediate variables, which might be a burden. However, some optimization algorithms are invertible, such as SGD with momentum (Maclaurin et al., 2015). In this case, no additional memory is needed.
Linear convergence: a case for the analytic estimator In the time it takes to compute $g_{t}^{2}$ , one can at the same cost compute $g_{c t}^{1}$ . If $z_{t}$ converges linearly at rate $κ^{t}$ , Proposition 1 shows that $g_{c t}^{1} - g^{*} = O (κ^{c t})$ , while Proposition 2 gives, at best, $g_{t}^{2} - g^{*} = O (κ^{2 t})$ : $g_{c t}^{1}$ is a better estimator of $g^{*}$ than $g_{t}^{2}$ , provided that $c \geq 2$ . In the quadratic case, we even have $g_{t}^{2} = g_{2 t}^{1}$ . Further, computing $g^{2}$ might requires additional memory: $g_{c t}^{1}$ should be preferred over $g_{t}^{2}$ in this setting. However, our analysis is only asymptotic, and other effects might come into play to tip the balance in favor of $g^{2}$ .

As it appears clearly in Table 1, choosing $g^{1}$ over $g^{3}$ depends on $t$ : when $m n t ≫ m^{3} + m^{2} n$ , the additional cost of computing $g^{3}$ is negligible, and it should be preferred since it is more accurate. This is however a rare situation in a large scale setting.
Sublinear convergence We have provided two settings where $z_{t}$ converges sub-linearly. In the stochastic gradient descent case with a fixed step-size, one can benefit from using $g^{2}$ over $g^{1}$ , since it allows to reach an accuracy that can never be reached by $g^{1}$ . With a decreasing step-size, reaching $| g_{t}^{1} - g^{*} | \leq ε$ requires $O (ε^{- 2 / α})$ iterations, while reaching $| g_{t}^{2} - g^{*} | \leq ε$ only takes $O (ε^{- 1 / α})$ iterations. For $ε$ small enough, we have $c ε^{- 1 / α} < ε^{- 2 / α}$ : it is always beneficial to use $g^{2}$ if memory capacity allows it.

The story is similar for the simple non-strongly convex problem studied in subsection 3.4: because of the slow convergence of the algorithms, $g_{t}^{2}$ is much closer to $g^{*}$ than $g_{c t}^{1}$ . Although our analysis was carried in the simple least mean $p$ -th problem, we conjecture it could be extended to the more general setting of $p$ -Łojasiewicz functions (Attouch and Bolte, 2009).

Figure 2: Evolution of $| g_{t}^{i} - g^{*} |$ with the number of iteration $t$ for (a
) the Ridge Regression
$L_{1}$ , (b
) the Regularized Logistic Regression
$L_{2}$ , (c) the Least Mean $p$ -th Norm $L_{3}$ in *log-scale* and (d) the Wasserstein Distance $L_{4}$ . In all cases, we can see the asymptotic super-efficiency of the $g_{2}$ estimator compared to $g_{1}$ . The $g_{3}$ estimator is better in most cases but it is unstable in (d).

5 Experiments

All experiments are performed in Python using pytorch (Paszke et al., 2019). The code to reproduce the figures is available online.¹¹1See Appendix.

5.1 Considered losses

In our experiments, we considered several losses with different properties. For each experiments, the details on the size of the problems are reported in subsection A.1.
Regression For a design matrix $D \in R^{n \times m}$ and a regularization parameter $λ > 0$ , we define

	$L_{1} (z, x) =$	$\frac{1}{2} \| x - D z \|^{2} + \frac{λ}{2} \| z \|^{2},$
	$L_{2} (z, x) =$	$n \sum i = 1 log (1 + e^{- x_{i} [D z]_{i}}) + \frac{λ}{2} \| z \|$

	$\| g_{t}^{1} (x) - g^{*} (x) \|$	$= O (\| z_{t} (x) - z^{*} (x) \|),$
	$\| g_{t}^{2} (x) - g^{*} (x) \|$	$= o (\| z_{t} (x) - z^{*} (x) \|),$
	$\| g_{t}^{3} (x) - g^{*} (x) \|$	$= O (\| z_{t} (x) - z^{*} (x) \|^{2}) .$

$\| g^{1} - g^{*} \|$	$\leq L_{x} \| z_{0} - z^{*} \| κ^{t},$
$\| g^{2} - g^{*} \|$	$\leq (ρ L_{z} t + \frac{κ}{2}) L \| z_{0} - z^{*} \|^{2} κ^{2 t - 1},$	(13)
$\| g^{3} - g^{*} \|$	$\leq (\frac{L}{2} + L_{J} L_{z}) \| z_{0} - z^{*} \|^{2} κ^{2 t} .$