SAGA and Restricted Strong Convexity

1 Introduction

We study the finite sum problem in the following forms:

Convex $G (θ)$ :

(1)

where $f_{i} (θ)$ is a convex loss such as $f_{i} (θ) = \frac{1}{2} (y_{i} - θ^{T} x_{i})^{2}$ and $ψ (θ)$ is a norm, $ρ$ is some predefined radius. We denote the dual norm of $ψ (θ)$ as $ψ^{*} (θ)$ and assume that each $f_{i} (θ)$ is $L$ smooth.
Non-convex $G (θ)$ :

$\begin{matrix} % Minimize: g_{λ} (θ) \leq ρ G (θ) & ≜ f (θ) + g_{λ, μ} (θ) = \frac{1}{n} n \sum i = 1 f_{i} (θ) + g_{λ, μ} (θ), \end{matrix}$ (2)

where $f_{i} (θ)$ is convex and $L$ smooth, $g_{λ, μ} (θ)$ is some non-convex regularizer, $g_{λ} (θ)$ is close related to $g_{λ, μ} (θ)$ and we defer the formal definition to the section 2.3.

Such finite sum structure is common for machine learning problems particularly in the empirical risk minimization (

ERM) setting. To solve the above problem, the standard Prox-full gradient (FG) method update

θ^{k}

iterative by

θ^{k + 1} = p r o x_{γ λ ψ} (θ^{k} - γ \nabla f (θ^{k})) .

It is well known that FG enjoys fast linear convergence under smoothness and strong convexity assumption. However this result may be less appealing when $n$ is large since the cost of calculation of full gradient scales with $n$ . Stochastic gradient (SG) method remedies this issue but only possess the sub-linear convergence rate.

Recently, a set of stochastic algorithms including SVRG Johnson and Zhang (2013); Xiao and Zhang (2014), SAGA Defazio et al. (2014), SAG Schmidt et al. (2013) SDCA Shalev-Shwartz and Zhang (2014) and many others Harikandeh et al. (2015); Qu et al. (2015); Zhang and Lin (2015) have been proposed to exploit the finite sum structure and enjoy linear rate convergence under smoothness and strong convexity assumption on $f_{i} (θ)$ . We study SAGA in this paper. From a high level, SAGA is a midpoint between SAG and SVRG, see the discuss in Defazio et al. (2014)

for more details. Different from SVRG, it is a fully incremental gradient method. Comparing with SAG, it uses an unbiased estimator of the gradient, which results in an easier proof among other things. In fact, to the best of our knowledge, the analysis of SAG has not yet been extended to proximal operator version.

A second trendy topic in optimization and statistical estimation is the study of non-convex problems, due to a vast array of applications such as SCAD Fan and Li (2001), MCP Zhang and Zhang (2012), robust regression (Corrected Lasso Loh and Wainwright (2011)

) and deep learning

Goodfellow et al. (2016). Some previous work have established fast convergence for batch gradient methods without assuming strong convexity or even convexity: Xiao and Zhang (2013) proposed a homotopy method to solve Lasso with RIP condition. Agarwal et al. (2010) analyzed the convergence rate of batched composite gradient method on several models, such as Lasso, logistic regression with

ℓ_{1}

regularization and noisy matrix decomposition, and showed that the convergence is linear under mild conditions of the solution (sparse or low rank). Loh and Wainwright (2011, 2013) extended the above work to the non-convex case.

These two line of research thus motivate this work to investigate whether SAGA enjoys the linear convergence rate without strong convexity or even in the non-convex problem. Specifically, we prove that under Restricted strong convexity assumption, SAGA converges linearly up to the fundamental statistical precision of the model, which covers five statistical models we mentioned above but not limited to these. In a high level, it is a stochastic counterpart of the work in Loh and Wainwright (2013), albeit with more involved analysis due to the stochastic nature of SAGA.

We list some notable non-strongly convex and non-convex problems in the following. Indeed, our work proves that SAGA converges linearly in all these models. Note that the first three belong to the non-strongly convex category especially when $p > n$ and the last two are non-convex.

Lasso: $f_{i} (θ) = \frac{1}{2} (⟨ θ, x_{i} ⟩ - y_{i})^{2}$ and $ψ (θ) = ∥ θ ∥_{1}$ .
Group Lasso: $f_{i} (θ) = \frac{1}{2} (⟨ θ, x_{i} ⟩ - y_{i})^{2}$ , $ψ (θ) = ∥ θ ∥_{1, 2}$ .
Logistic Regression with $l_{1}$ regularization: $f_{i} (θ) = log (1 + exp (- y_{i} ⟨ x_{i}, θ ⟩)$ and $ψ (θ) = ∥ θ ∥_{1}$ .
Corrected Lasso Loh and Wainwright (2011): $G (θ) = \sum_{i = 1}^{n} \frac{1}{2 n} (⟨ θ, x_{i} ⟩ - y_{i})^{2} - \frac{1}{2} θ^{T} Σ θ + λ ∥ θ ∥_{1},$ where $Σ$ is some positive definite matrix.
Regression with SCAD regularizer Fan and Li (2001): $G (θ) = \sum_{i = 1}^{n} \frac{1}{2 n} (⟨ θ, x_{i} ⟩ - y_{i})^{2} + S C A D (θ)$ .

Very recently, Qu et al. (2016) explore the similar idea of us called restrict strong convexity condition (RSC) Negahban et al. (2009) on SVRG and prove that under this condition, a class of ERM

problem has the linear convergence even without strongly convex or even the convex assumption. From a high level perspective, our work can be thought as of similar spirit but for SAGA algorithm. We believe analyzing the SAGA algorithm is indeed important as SAGA enjoys certain advantage compared to SVRG. As discussed above, SVRG is not a completely incremental algorithm since it need to calculate the full gradient in every epoch, while SAGA avoids the computation of the full gradient by keeping a table of gradient. Moreover, although in general SAGA costs

O (n p)

storage (which is inferior to SVRG), in many scenarios the requirement of storage can be reduced to

O (n)

. For example, many loss function

f_{i}

take the form

f_{i} (θ) = g_{i} (θ^{T} x_{i})

for a vector

x_{i}

and since

x_{i}

is a constant we just need to store the scalar

\nabla g_{i_{k}} (u_{i}^{k})

for

u_{i}^{k} = x_{i_{k}}^{T} θ^{k}

rather than full gradient. When this scenario is possible, SAGA can perform similarly or even better than SVRG. In addition, SVRG has an additional parameter besides step size to tune – the number of iteration

m

per inner loop. To conclude, both SVRG and SAGA can be more suitable for some problems, and hence it is useful to understand the performance of SAGA for non-strongly convex or non-convex setups. At last, the proof steps are very different. In particular, we define a Lyapunov function in SAGA and prove it converges geometrically until the optimality gap achieves the statistical tolerance, while Qu et al. (2016) directly look at evolution of

G (θ^{k})

1.1 Related work

There are a plethora of work on the finite sum problem and we review those most closely related to ours. Li et al. (2016) consider SVRG on a non-convex sparse linear regression setting different from ours, where $f_{i}$ is convex and the non-convexity comes from the hard-thresholding operator. We focus on a non-convex regularizer such as SCAD and corrected Lasso. In addition, we consider a unified framework on SAGA thus our work not only covers the linear sparse model but also the group sparsity and other model satisfying our assumptions. Karimi et al. (2016); Reddi et al. (2016); Hajinezhad et al. (2016) proved global linear convergence of SVRG and SAGA on non-convex problems by revisiting the concept Polyak-Łojasiewicz inequality or its equivalent idea such as error bound . We emphasize that our work looks at the problem from different perspective. In particular, our theory asserts that the algorithm converges faster with sparser $θ^{*}$ , while their results are independent of the sparsity $r$ . Empirical observation seems to agree with our theorem. Indeed, when $r$

is dense enough a phase transition from linear rate to sublinear rate occurs (also observed in

Qu et al. (2016)), which agrees with the prediction of our theorem. Furthermore, their work requires the epigraph of

ψ (θ)

to be a polyhedral set which limits its applicability. For instance, the popular group Lasso does not satisfy such an assumption. Other non-convex stochastic variance reduction works include

Shalev-Shwartz (2016); Shamir (2015) and Allen-Zhu and Hazan (2016): Shalev-Shwartz (2016) considers the setting that

f (θ)

is strongly convex but each individual

f_{i} (θ)

is non-convex. Shamir (2015) discusses a projection version of non-convex SVRG and its specific application on PCA. Allen-Zhu and Hazan (2016) consider a general non-convex problem, which only achieves a sublinear convergence rate.

2 Preliminaries

2.1 Restricted Strong Convexity

As mentioned in the abstract, Restricted strong convexity (RSC) is the key assumption underlying our results. We therefore define RSC formally. We say a function $f (θ)$ satisfies RSC w.r.t. to a norm $ψ (θ)$ with parameter $(σ, τ_{σ})$ if the following holds.

\begin{matrix} f (θ_{2}) - f (θ_{1}) - ⟨ \nabla f (θ_{2}), θ_{2} - θ_{1} ⟩ \geq & \frac{σ}{2} ∥ θ_{2} - θ_{1} ∥_{2}^{2} - τ_{σ} ψ^{2} (θ_{2} - θ_{1}) . \end{matrix}

(3)

We remark that we assume $f (θ) = \frac{1}{n} \sum_{i = 1}^{n} f_{i} (θ)$ satisfies the RSC rather than individual loss function $f_{i} (θ)$ . Indeed, $f_{i} (θ)$ does not satisfy RSC in practice. Note that when $f (θ)$ is $σ -$ strongly convex, obviously we have $τ_{σ} = 0$ . For more discussions on RSC, we refer reader to Negahban et al. (2009).

2.2 Assumptions for the Convex regularizer $Ψ (θ)$

2.2.1 Decomposibility of $Ψ (θ)$

Given a pair of subspaces $M \subseteq ¯ M$ in $R^{p}$ , the orthogonal complement of $¯ M$ is

{¯ M}^{⊥} = {v \in R^{p} | ⟨ u, v ⟩ = 0 ~{}for all~{} u \in ¯ M} .

$M$ is known as the model subspace, where ${¯ M}^{⊥}$ is called the perturbation subspace, representing the deviation from the model subspace. A regularizer $ψ$ is decomposable w.r.t. $(M, {¯ M}^{⊥})$ if

ψ (θ + β) = ψ (θ) + ψ (β)

for all $θ \in M$ and $β \in {¯ M}^{⊥} .$ A concrete example is $ℓ_{1}$ regularization for sparse vector supported on subset $S$ . We define the subspace pairs with respect to the subset $S \subset {1, . . ., p}$ , $M (S) = {θ \in R^{p} | θ_{j} = 0 ~{}for all~{} j \notin S}$ and $¯ M (S) = M (S) .$ The decomposability is thus easy to verify. Other widely used examples include non-overlap group norms such as $∥ \cdot ∥_{1, 2}$ , and the nuclear norm $∥ | \cdot ∥ |_{*}$ Negahban et al. (2009). In the rest of the paper, we denote $θ_{M}$ as the projection of $θ$ on the subspace $M$ .

2.2.2 Subspace compatibility

Given the regularizer $ψ (\cdot)$ , the subspace compatibility $H (¯ M)$ is given by

H (¯ M) = sup θ \in ¯ M ∖ {0} \frac{ψ (θ)}{∥ θ ∥_{2}} .

In other words, it is the Lipschitz constant of the regularizer restricted in $¯ M .$ For instance, in the above-mentioned sparse vector example with cardinality $r$ , $H (¯ M) = \sqrt{r}$ .

2.3 Assumptions for the Nonconvex regularizer $g_{λ, μ} (θ)$

In the non-convex case, we consider regularizers that are separable across coordinates, i.e., $g_{λ, μ} (θ) = \sum_{j = 1}^{p} {¯ g}_{λ, μ} (θ_{j})$ . Besides the separability, we have additional assumptions on $g_{λ, μ} (\cdot)$ . For the univariate function ${¯ g}_{λ, μ} (t)$ , we assume

${¯ g}_{λ, μ} (\cdot)$ satisfies ${¯ g}_{λ, μ} (0) = 0$ and is symmetric around zero. That is, ${¯ g}_{λ, μ} (t) = {¯ g}_{λ, μ} (- t)$ .
On the nonnegative real line, ${¯ g}_{λ, μ}$ is nondecreasing.
For $t > 0$ , $\frac{{¯ g}_{λ, μ} (t)}{t}$ is nonincreasing in t.
${¯ g}_{λ, μ} (t)$ is differentiable at all $t \neq 0$ and subdifferentiable at $t = 0$ , with ${lim}_{t \to 0^{+}} g_{λ, μ}^{'} (t) = λ L_{g}$ for a constant $L_{g}$ .
${¯ g}_{λ} (t) := ({¯ g}_{λ, μ} (t) + \frac{μ}{2} t^{2}) / λ$ is convex.

We provide two examples satisfying the above assumptions.

$(1) S C A D_{λ, ζ} (t) ≜$

⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} λ | t |, & for | t | \leq λ, - (t^{2} - 2 ζ λ | t | + λ^{2}) / (2 (ζ - 1)), & for λ < | t | \leq ζ λ, (ζ + 1) λ^{2} / 2, & for | t | > ζ λ, \end{matrix}

where $ζ > 2$ is a fixed parameter. It satisfies the assumption with $L_{g} = 1$ and $μ = \frac{1}{ζ - 1}$ Loh and Wainwright (2013).

(2) M C P_{λ, b} (t) ≜ sign (t) λ \int_{0}^{| t |} (1 - \frac{z}{λ b})_{+} d z,

where $b > 0$ is a fixed parameter. MCP satisfies the assumption with $L_{g} = 1$ and $μ = \frac{1}{b}$ Loh and Wainwright (2013).

Input: Step size

γ

, number of iterations

K

, and smoothness parameters

L

for

k = 1, . . ., K

Pick a

j

uniformly at random

1. Take

ϕ_{j}^{k + 1} = θ^{k}

, and store

\nabla f_{j} (ϕ_{j}^{k + 1})

in the table. All other entries in the table remain unchanged.

2. Update

θ

using

\nabla f_{j} (ϕ_{j}^{k + 1})

\nabla f_{j} (ϕ_{j}^{k})

and the table average:

w^{k + 1} = θ^{k} - γ [\nabla f_{j} (ϕ_{j}^{k + 1}) - \nabla f_{j} (ϕ_{j}^{k}) + \frac{1}{n} n \sum i = 1 \nabla f_{i} (ϕ_{i}^{k})] .

θ^{k + 1} = arg {min}_{ψ (θ) \leq ρ} \frac{1}{2} ∥ θ - w^{k + 1} ∥_{2}^{2} + γ λ ψ (θ) .

end for

Algorithm 1 SAGA

2.4 Implementation of the algorithm

For the convex $G (θ)$ case, we directly apply the Algorithm 1. As to the non-convex $G (θ)$ case, we essentially solve the following equivalent problem

Minimize: g_{λ} (θ) \leq ρ (f (θ) - \frac{μ}{2} ∥ θ ∥_{2}^{2}) + λ g_{λ} (θ) .

We define $F_{i} (θ) = f_{i} (θ) - \frac{μ}{2} ∥ θ ∥_{2}^{2}$ and $F (θ) = f (θ) - \frac{μ}{2} ∥ θ ∥_{2}^{2}$ . To implement Algorithm 1 on non-convex $G (θ)$ , we replace $f_{i} (\cdot)$ and $ψ (\cdot)$ in the algorithm by $F_{i} (\cdot)$ and $g_{λ} (\cdot)$ . Remark that according to the assumptions on $g_{λ, μ} (\cdot)$ in Section 2.3, $g_{λ} (\cdot)$ is convex thus the proximal step is well-defined. The update rule of proximal operator on several $g_{λ, μ}$ (such as SCAD) can be found in Loh and Wainwright (2013) .

3 Main result

In this section, we present the main theoretical results, and some corollaries that instantiate the main results in several well known statistical models.

3.1 Convex $G (θ)$

We first present the results on convex $G (θ)$ . In particular, we prove a Lyapunov function converges geometrically until $G (θ^{k}) - G (^θ)$ achieves some tolerance. To this end, we first define the Lyapunov function

\begin{matrix} T_{k} ≜ \frac{1}{n} n \sum i = 1 (f_{i} (ϕ_{i}^{k}) - f_{i} (^θ) - ⟨ \nabla f_{i} (^θ), ϕ_{i}^{k} -^θ ⟩) + (c + α) ∥ θ^{k} -^θ ∥_{2}^{2} + b (G (θ^{k}) - G (^θ)), \end{matrix}

where $^θ$ is the optimal solution of problem (1), $c$ , $α$ , $b$ are some positive constant will be specified later in the theorems. Notice our definition is a little different from the one in the original SAGA paper in Defazio et al. (2014). In particular, we have an additional term $G (θ^{k}) - G (^θ)$ and choose different value of $c$ and $α$ , which helps us to utilize the idea of RSC.

We list some notations used in the following theorems and corollaries.

$θ^{*}$ is the unknown true parameter. $^θ$ is the optimal solution of (1).
$ψ^{*} (\cdot)$ is the dual norm of $ψ (\cdot)$ .
Modified restricted strongly convex parameter:

$¯ σ = σ - 64 τ_{σ} H^{2} (¯ M) .$
Tolerance

$δ = 24 τ_{σ} {(8 H (¯ M) ∥^θ - θ^{*} ∥_{2} + 8 ψ (θ_{M^{⊥}}^{*}))}_{2}^{2}$

Theorem 1.

Assume each $f_{i} (θ)$ is $L$ smooth and convex, $f (θ)$ satisfies the RSC condition with parameter $(σ, τ_{σ})$ and $¯ σ > 0$ , $θ^{*}$ is feasible, the regularizer is decomposable w.r.t. $(M, ¯ M)$ , if we choose the parameter $λ \geq max (2 ψ^{*} (\nabla f (θ^{*})), c_{1} τ_{σ} ρ)$ where $c_{1}$ is some universal positive constant, then with $γ = \frac{1}{9 L}$ , $c = \frac{9 L}{n}$ , $α = \frac{c}{2}$ , $b = 2 α γ$ , $\frac{1}{κ} = min (\frac{¯ σ}{14 L}, \frac{1}{9 n})$ , we have

E T_{k} \leq {(1 - \frac{1}{κ})}^{k} T_{0},

until $G (θ^{k}) - G (^θ) \leq δ,$ where the expectation is for the randomness of sampling of $j$ in the algorithm.

Some remarks are in order.

The requirement $¯ σ > 0$ is easy to satisfy in some popular statistical models. Take Lasso as an example, where $τ_{σ} = c_{2} \frac{log p}{n}$ , $c_{2}$ are some positive constant, $H^{2} (¯ M) = r$ . Thus $¯ σ = σ - 64 c_{2} \frac{r log p}{n}$ . Hence when $64 c_{2} \frac{r log p}{n} \leq \frac{1}{2} σ$ , we have $¯ σ \geq \frac{σ}{2}$ .
Since $\frac{1}{κ}$ depends on $¯ σ / L$ , the convergence rate is indeed affected by the sparsity $r$ (Lasso for example )as we mentioned in the introduction. Particularly, sparser $r$ leads to larger $¯ σ$ and faster convergence rate.
In some models, we can choose the subspace pair such that $θ^{*} \in M$ , thus the tolerance $δ$ is simplified to $δ = c_{3} τ_{σ} H^{2} (¯ M) ∥^θ - θ^{*} ∥_{2}^{2}$ . In Lasso as we mentioned above, $δ = c_{3} \frac{r log p}{n} ∥^θ - θ^{*} ∥_{2}^{2},$ i.e., the tolerance is dominated by the statistical error $∥^θ - θ^{*} ∥_{2}^{2} .$
When $G (θ^{k}) - G (^θ) \leq δ$ , use modified restricted strong convexity (Lemma 5 in the appendix), it is easy to derive $∥ θ^{k} -^θ ∥_{2}^{2} \leq \frac{c_{4} δ}{¯ σ} .$

Combine all remarks together, the theorem says the Lyapunov function decreases geometrically until $G (θ^{k}) - G (^θ)$ achieves the tolerance $δ$ . This tolerance is dominated by the statistical error $∥ θ^{k} -^θ ∥_{2}^{2}$ , thus can be ignored from the statistical perspective.

3.1.1 Sparse linear regression

The first model we consider is Lasso, where $f_{i} (θ) = \frac{1}{2} (⟨ θ, x_{i} ⟩ - y_{i})^{2}$ and $ψ (θ) = ∥ θ ∥_{1}$ . More concretely, we consider a model where each data point $x_{i}$

is i.i.d. sampled from a zero-mean normal distribution, i.e.,

x_{i} \sim N (0, Σ)

. We denote the data matrix by

X \in R^{n \times p}

and the smallest eigenvalue of

Σ

σ_{min} (Σ)

, and let

ν (Σ) ≜ {max}_{i = 1, . . ., p} Σ_{i i}

. The observation is generated by

y_{i} = x_{i}^{T} θ^{*} + ξ_{i}

, where

ξ_{i}

is a zero mean sub-Gaussian noise with variance

ς^{2}

. We use

X_{j} \in R^{n}

to denote

j

-th column of

X

. Without loss of generality, we require

X

to be column-normalized, i.e.,

\frac{∥ X_{j} ∥_{2}}{\sqrt{n}} \leq 1 for all j = 1, 2, . . ., p

. Here, the constant

1

is chosen arbitrarily to simplify the exposition, as we can always rescale the data.

Corollary 1.

Assume $θ^{*}$ is the true parameter supported on a subset with cardinality at most $r$ , and we choose $λ$ such that $λ \geq max (6 ς \sqrt{\frac{log p}{n}}, c_{1} ρ ν (Σ) \frac{log p}{n})$ , $¯ σ = \frac{1}{2} σ_{min} (Σ) - c_{2} ν (Σ) \frac{r log p}{n}$ , then with $γ = \frac{1}{9 L}$ , $c = \frac{9 L}{n}$ , $α = \frac{c}{2}$ , $b = 2 α γ$ , $\frac{1}{κ} = min (\frac{¯ σ}{14 L}, \frac{1}{9 n})$ , we have

E T_{k} \leq {(1 - \frac{1}{κ})}^{k} T_{0},

with probability at least

1 - exp (- 3 log p) - exp (- c_{3} n)

, until

G (θ^{k}) - G (^θ) \leq δ,

where

δ = c_{4} ν (Σ) \frac{r log p}{n} ∥^θ - θ^{*} ∥_{2}^{2} .

Here

c_{1}, c_{2}, c_{3}, c_{4}

are some universal positive constants.

We offer some discussions on this corollary.

The requirement of $λ \geq 6 ς \sqrt{\frac{log p}{n}}$ is known to play an important role in proving bounds on the statistical error of Lasso, see Negahban et al. (2009) and reference therein for further details.
The requirement $λ \geq c_{1} ρ ν (Σ) \frac{log p}{n}$ is to guarantee the fast global convergence of the algorithm, which is similar to the requirement in its batch counterpart Agarwal et al. (2010).
When $r$ is small and $n$ is large, which is necessary for statistical consistency of Lasso, we obtain $¯ σ > 0$ , which guarantees the existences of $κ$ . Under this condition we have $δ = c_{4} ν (Σ) \frac{r log p}{n} ∥^θ - θ^{*} ∥_{2}^{2}$ , which is dominated by $∥^θ - θ^{*} ∥_{2}^{2} .$

3.1.2 Group Sparse model

The group sparsity model aims to find a regressors such that predefined groups of covariates are selected into or out of a model together. The most commonly used regularization to encourage group sparsity is $∥ \cdot ∥_{1, 2}$ . Formally, we are given a class of disjoint groups of the features, i.e., $G = {G_{1}, G_{2}, . . ., G_{N_{G}}}$ and $G_{i} \cap G_{j} = \emptyset$ . The regularization term is $∥ θ ∥_{G, q} ≜ \sum_{g = 1}^{N_{g}} ∥ θ_{g} ∥_{q}$ . When $q = 2$ , it reduces to the popular group Lasso Yuan and Lin (2006) while another widely used case is $q = \infty$ Turlach et al. (2005); Quattoni et al. (2009).

We now define the subspace pair $(M, ¯ M)$ in the group sparsity model. For a subset $S_{G} \subseteq {1, . . ., N_{G}}$ with cardinality $s_{G} = | S_{G} |$ , we define the subspace

M (S_{G}) = {θ | θ_{G_{i}} = 0 for all i \notin S_{G}},

and $M = ¯ M$ . The orthogonal complement is

{¯ M}^{⊥} (S_{G}) = {θ | θ_{G_{i}} = 0 for all % i \in S_{G}} .

We can easily verify that

∥ α + β ∥_{G, q} = ∥ α ∥_{G, q} + ∥ β ∥_{G, q},

for any $α \in M (S_{G})$ and $β \in {¯ M}^{⊥} (S_{G})$ .

We mainly focus on the discussion on the case $q = 2$ , i.e., group Lasso. We require the following condition, which generalizes the column normalization condition in the Lasso case. Given a group $G$ of size $m$ and $X_{G} \in R^{n \times m}$ , the associated operator norm $| | | X_{G_{i}} | | |_{q \to 2} ≜ {max}_{∥ θ ∥_{q} = 1} ∥ X_{G} θ ∥_{2}$ satisfies

\frac{| | | X_{G_{i}} | | |_{q \to 2}}{\sqrt{n}} \leq 1 for all i = 1, 2, . . ., N_{G} .

The condition reduces to the column normalized condition when each group contains only one feature (i.e., Lasso).

In the following corollary, we use $q = 2$ , i.e., group Lasso, as an example. We assume the observation $y_{i}$ is generated by $y_{i} = x_{i}^{T} θ^{*} + ξ_{i}$ , where $x_{i} \sim N (0, Σ)$ , and $ξ_{i} \sim N (0, ς^{2})$ .

Corollary 2.

(Group Lasso) Assume $θ \in R^{p}$ and each group has $m$ parameters, i.e., $p = m N_{G}$ . Denote the cardinality of non-zero group by $s_{G}$ , and we choose parameter $λ$ such that

λ \geq max (4 ς (\sqrt{\frac{m}{n}} + \sqrt{\frac{log N_{G}}{n}}), c_{1} ρ σ_{2} (Σ) (\sqrt{\frac{m}{n}} + \sqrt{\frac{3 log N_{G}}{n}})^{2}),

then with $γ = \frac{1}{9 L}$ , $c = \frac{9 L}{n}$ , $α = \frac{c}{2}$ , $b = 2 α γ$ , $\frac{1}{κ} = min (\frac{¯ σ}{14 L}, \frac{1}{9 n})$ , we have

E T_{k} \leq (1 - \frac{1}{κ})^{k} T_{0}

with probability at least $1 - 2 exp (- 2 log N_{G}) - c_{2} exp (- c_{3} n)$ , until $G (θ^{k}) - G (^θ) \leq δ,$ where $¯ σ = σ_{1} (Σ) - c_{2} σ_{2} (Σ) s_{G} (\sqrt{\frac{m}{n}} + \sqrt{\frac{3 log N_{G}}{n}})^{2}$ , $σ_{1} (Σ)$ and $σ_{2} (Σ)$ are positive constant depending only on $Σ$ , $δ = c_{4} σ_{2} (Σ) s_{G} (\sqrt{\frac{m}{n}} + \sqrt{\frac{3 log N_{G}}{n}})^{2} ∥^θ - θ^{*} ∥_{2}^{2}$ . $c_{1}, c_{2}, c_{3}, c_{4}$ are some universal positive constants.

We offer some discussions to put above corollary into context.

To satisfy the requirement of $¯ σ > 0$ , it suffices to have $s_{G} (\sqrt{\frac{m}{n}} + \sqrt{\frac{3 log N_{G}}{n}})^{2} = o (1)$ . It is also the condition to guarantee the statistical consistency of group Lasso Negahban et al. (2009).
$s_{G}$ and $m$ affect the speed of the convergence, in particular, smaller $m$ and $s_{G}$ leads to larger $¯ σ$ and thus $¯ σ / L$ .
The requirement of $λ$ is similar to the batch gradient method in Agarwal et al. (2010).

3.2 Non-convex $G (θ)$

The definition of Lyapunov function in the non-convex case is same with the convex one, i.e.,

\begin{matrix} T_{k} ≜ \frac{1}{n} n \sum i = 1 (f_{i} (ϕ_{i}^{k}) - f_{i} (^θ) - ⟨ \nabla f_{i} (^θ), ϕ_{i}^{k} -^θ ⟩) + (c + α) ∥ θ^{k} -^θ ∥_{2}^{2} + b (G (θ^{k}) - G (^θ)) . \end{matrix}

Note that $^θ$ is the global optimum of problem (2) and $f_{i} (\cdot)$ is convex, thus $T_{k}$ is always positive. In the non-convex case, we require $f (θ)$ satisfy the RSC condition with parameter $(σ, τ \frac{log p}{n})$ , where $τ$ is some positive constant.

We list some notations used in the following theorem and corollaries of it.

$^θ$ is the global optimum of problem (2), and $θ^{*}$ is the unknown true parameter with cardinality $r$ .
Modified restricted strongly convex parameter:

$¯ σ = σ - 64 r τ \frac{log p}{n} - μ .$

Recall $μ$ is defined in section 2.3 and represent the degree of non-convexity.
Tolerance $δ = c_{1} r τ \frac{log p}{n} ∥^θ - θ^{*} ∥_{2}^{2}$ , where $c_{1}$ is some universal positive constant.

Theorem 2.

Suppose $θ^{*}$ is $r$ sparse, $^θ$ is the global optimum of Problem (2), each $f_{i} (θ)$ is L smooth and convex, $f (θ)$ satisfies the RSC condition with $(σ, τ \frac{log p}{n})$ , $¯ σ > 3 μ$ , $L > 3 μ$ , $g_{λ, μ}$ satisfies the assumption in Section 2.3, and $λ L_{g} \geq max {c_{1} ρ τ \frac{log p}{n}, 4 ∥ \nabla f (θ^{*}) ∥_{\infty}}$ , where $c_{1}$ is some positive constant, then with $γ = \frac{1}{24 L}$ , $c = \frac{24 L}{n}$ , $α = \frac{c}{2}$ , $b = 2 α γ$ , $\frac{1}{κ} = \frac{1}{24} min (\frac{2 ¯ σ}{5 L}, \frac{1}{n})$ , we have

E T_{k} \leq {(1 - \frac{1}{κ})}^{k} T_{0},

until $G (θ^{k}) - G (^θ) \leq δ,$ where the expectation is for the randomness of sampling of $j$ in the algorithm.

Notice that we require $¯ σ > 3 μ$ , that is $σ - 64 r τ \frac{log p}{n} - 4 μ > 0$ . Thus to satisfy this requirement, the non-convex parameter $μ$ can not be large.
The tolerance $δ = c_{2} r τ \frac{log p}{n} ∥^θ - θ^{*} ∥_{2}^{2}$ is dominated by the statistical error $∥^θ - θ^{*} ∥_{2}^{2}$ , when the model is sparse ( $r$ is small ) and $n$ is large.
When $G (θ^{k}) - G (^θ) \leq δ$ , using the modified restricted strong convexity on non-convex $G (θ)$ (Lemma 10 in the appendix), we obtain $∥ θ^{k} -^θ ∥_{2}^{2} \leq c_{3} \frac{δ}{¯ σ} .$
The requirement of $λ$ is similar to the batched gradient algorithm Loh and Wainwright (2013).

Again, the theorem says the Lyapunov function decreases geometrically until $G (θ^{k}) - G (^θ)$ achieves the tolerance $δ$ and this tolerance can be ignored from the statistical perspective.

3.2.1 Linear regression with SCAD regularization

The first non-convex model we considered is linear regression with SCAD regularization. The loss function is $f_{i} (w) = \frac{1}{2} (y - ⟨ θ, x_{i} ⟩)^{2}$ , and $g_{λ, μ} (\cdot)$ is $S C A D (\cdot)$ with parameter $λ$ and $ζ$ . The data $(x_{i}, y_{i})$ are generated in the similar way as that in Lasso case.

Corollary 3.

(Linear regression with SCAD regularization) Suppose $θ^{*}$ is the true parameter supported on a subset with cardinality at most $r$ , $^θ$ is the global optimum, $¯ σ \geq \frac{3}{ζ - 1}$ , $L > \frac{3}{ζ - 1}$ and we choose $λ$ such that $λ \geq max {c_{1} ρ ν (Σ) \frac{log p}{n}, 12 ς \sqrt{\frac{log p}{n}}}$ then with $γ = \frac{1}{24 L}$ , $c = \frac{24 L}{n}$ , $α = \frac{c}{2}$ , $b = 2 α γ$ , $\frac{1}{κ} = \frac{1}{24} min (\frac{2 ¯ σ}{5 L}, \frac{1}{n})$ , we have

E T_{k} \leq {(1 - \frac{1}{κ})}^{k} T_{0},

with probability at least $1 - exp (- 3 log p) - exp (- c_{2} n)$ , until $G (θ^{k}) - G (^θ) \leq δ,$ where $¯ σ = \frac{1}{2} σ_{min} (Σ) - c_{3} ν (Σ) \frac{r log p}{n} - \frac{1}{ζ - 1}$ , $δ = c_{4} ν (Σ) \frac{r log p}{n} ∥^θ - θ^{*} ∥_{2}^{2} .$ Here $c_{1}, c_{2}, c_{3}, c_{4}$ are some universal positive constants.

We remark that to satisfy the requirement $¯ σ \geq \frac{3}{ζ - 1}$ , we need the non-convex parameter $μ = \frac{1}{ζ - 1}$ to be small, the model sparse (r is small) and the number of sample $n$ large.

3.2.2 Linear regression with noisy covariates

The corrected Lasso is proposed by Loh and Wainwright (2011). Suppose data are generated according to a linear model $y_{i} = x_{i}^{T} θ^{*} + ξ_{i},$ where $ξ_{i}$ is a random zero-mean sub-Gaussian noise with variance $ς^{2} .$ The observation $z_{i}$ of $x_{i}$ is corrupted by addictive noise, in particular, $z_{i} = x_{i} + w_{i}$ , where $w_{i} \in R^{p}$ is a random vector independent of $x_{i}$ , with zero-mean and known covariance matrix $Σ_{w}$ . Define $^Γ = \frac{Z^{T} Z}{n} - Σ_{w}$ and $^γ = \frac{Z^{T} y}{n}$ . Our goal is to estimate $θ^{*}$ based on $y_{i}$ and $z_{i}$ (but not $x_{i}$ which is not observable), and the corrected Lasso proposes to solve the following:

^θ \in arg min ∥ θ ∥_{1} \leq ρ \frac{1}{2} θ^{T}^Γ θ -^γ θ + λ ∥ θ ∥_{1} .

Equivalently, it solves

min ∥ θ ∥_{1} \leq ρ \frac{1}{2 n} n \sum i = 1 (y_{i} - θ^{T} z_{i})^{2} - \frac{1}{2} θ^{T} Σ_{w} θ + λ ∥ θ ∥_{1} .

Notice that due to the term $- \frac{1}{2} θ^{T} Σ_{w} θ$ , the optimization problem is non-convex.

3.3 Corrected Lasso

We consider a model where each data point $x_{i}$ is i.i.d. sampled from a zero-mean normal distribution, i.e., $x_{i} \sim N (0, Σ)$ . We denote the data matrix by $X \in R^{n \times p}$ , the smallest eigenvalue of $Σ$ by $σ_{min} (Σ)$ and the largest eigenvalue by $σ_{max} (Σ)$ and let $ν (Σ) ≜ {max}_{i = 1, . . ., p} Σ_{i i}$ . We observe $z_{i}$ which is $x_{i}$ corrupted by addictive noise, i.e., $z_{i} = x_{i} + w_{i}$ , where $w_{i} \in R^{p}$ is a random vector independent of $x_{i}$ , with zero-mean and known covariance matrix $Σ_{w}$ .

Corollary 4.

(Corrected Lasso) Suppose we are given i.i.d. observations ${(z_{i}, y_{i})}$ from the linear model with additive noise, $θ^{*}$ is $r$ sparse and $Σ_{w} = γ_{w} I$ , $¯ σ > 3 γ_{w}$ , $L > 3 γ_{w}$ where $¯ σ = \frac{1}{2} σ_{min} (Σ) - c_{1} σ_{min} (Σ) max ((\frac{σ_{max} (Σ) + γ_{w}}{σ_{min} (Σ)})^{2}, 1) \frac{r log p}{n} - γ_{w}$ . Let $^θ$ be the global optimum. We choose $λ \geq max {c_{2} ρ \frac{log p}{n}, c_{3} φ \sqrt{\frac{log p}{n}}}$ where $φ = (\sqrt{σ_{max} (Σ)} + \sqrt{γ_{w}}) (ς + \sqrt{γ_{w}} ∥ θ^{*} ∥_{2})$ , then with $γ = \frac{1}{24 L}$ , $c = \frac{24 L}{n}$ , $α = \frac{c}{2}$ , $b = 2 α γ$ , $\frac{1}{κ} = \frac{1}{24} min (\frac{2 ¯ σ}{5 L}, \frac{1}{n})$ , we have

E T_{k} \leq (1 - \frac{1}{κ})^{k} T_{0},

with high probability at least $1 - c_{4} exp (- c_{5} n min (\frac{σ_{min}^{2} (Σ)}{(σ_{max} (Σ) + γ_{w})^{2}}, 1)) - exp (- c_{6} log p)$ until $G (θ^{k}) - G (^θ) \leq δ,$ where $δ = c_{7} σ_{min} (Σ) max ((\frac{σ_{max} (Σ) + γ_{w}}{σ_{min} (Σ)})^{2}, 1) \frac{r log p}{n} ∥^θ - θ^{*} ∥_{2}^{2} .$ $c_{1}$ to $c_{7}$ are some universal positive constants.

Some remarks are listed below.

The result can be easily extended to more general $Σ_{w} ⪯ γ_{w} I .$
To satisfy the requirement $¯ σ > 3 γ_{w}$ , we need

$γ \leq \frac{1}{4} (\frac{1}{2} σ_{min} (Σ) - c_{1} σ_{min} (Σ) max ((\frac{σ_{max} (Σ) + γ_{w}}{σ_{min} (Σ)})^{2}, 1) \frac{r log p}{n}) .$

Similar requirement is needed in the batch gradient method Loh and Wainwright (2013).
The requirement of $λ$ is similar to that in batch gradient method Loh and Wainwright (2013).

3.4 Extension to Generalized linear model

The results on Lasso and group Lasso are readily extended to generalized linear models, where we consider the model

^θ=argminθ∈Ω′{1nn∑i=1Φ(θ,xi)−yi⟨θ,xi⟩+λ∥θ∥1},

with $Ω^{'} = Ω \cap B_{2} (R)$ and $Ω = {θ | ∥ θ ∥_{1} \leq ρ}$ , where $R$ is a universal constant Loh and Wainwright (2013). This requirement is essential, for instance for the logistic function , the Hessian function $Φ^{''} (t) = \frac{exp (t)}{(1 + exp (t))^{2}}$ approached to zero as its argument diverges. Notice that when $Φ (t) = t^{2} / 2$ , the problem reduces to Lasso. The RSC condition admit the form

\frac{1}{n} n \sum i = 1 Φ^{''} (⟨ θ_{t}, x_{i} ⟩) (⟨ x_{i}, θ - θ^{'} ⟩)^{2} \geq \frac{σ}{2} ∥ θ - θ^{'} ∥_{2}^{2} - τ_{σ} ∥ θ - θ^{'} ∥_{1}, % for all θ, θ^{'} \in Ω^{'}

For a board class of log-linear models, the RSC condition holds with $τ_{σ} = c \frac{log p}{n}$ . Therefore, we obtain same results as those of Lasso, modulus change of constants. For more details of RSC conditions in generalized linear model, we refer the readers to Negahban et al. (2009).

4 Empirical Result

We report the experimental results in this section to validate our theorem that SAGA can enjoys the linear convergence rate without strong convexity or even without convexity. We did experiment both on synthetic and real datasets and compare SAGA with several candidate algorithms. The experiment setup is similar to Qu et al. (2016). Due to space constraints, some addition simulation results are presented in the appendix. The algorithms tested are Prox-SVRG Xiao and Zhang (2014), Prox-SAG which is a proximal version of the algorithm in Schmidt et al. (2013), proximal stochastic gradient (Prox-SGD), regularized dual averaging method (RDA) Xiao (2010) and the proximal full gradient method (Prox-GD) Nesterov (2013). For the algorithms with a constant learning rate (i.e., SAGA,Prox-SAG, Prox-SVRG, Prox-GD), we tune the learning rate from an exponential grid ${2, \frac{2}{2^{1}}, . . ., \frac{2}{2^{12}}}$ and chose the one with best performance. Below are some remarks on the candidate algorithms.

The linear convergence of SVRG in our setting has been proved in Qu et al. (2016).
We adapt SAG to its Prox version. To the best of our knowledge, the convergence of Prox-SAG has not been established. In addition, it is not known whether the Prox-SAG converges or not although it works well in the experiment.
The step size in Prox-SGD is $η_{k} = η$

SAGA and Restricted Strong Convexity

Authors

Linear convergence of SDCA in statistical estimation

Linear Convergence of SVRG in Statistical Estimation

A Unified Convergence Analysis for Shuffling-Type Gradient Methods

Non-convex Global Minimization and False Discovery Rate Control for the TREX

Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives

A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives

Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression

1 Introduction

1.1 Related work

2 Preliminaries

2.1 Restricted Strong Convexity

2.2 Assumptions for the Convex regularizer $Ψ (θ)$

2.2.1 Decomposibility of $Ψ (θ)$

2.2.2 Subspace compatibility

2.3 Assumptions for the Nonconvex regularizer $g_{λ, μ} (θ)$

2.4 Implementation of the algorithm

3 Main result

3.1 Convex $G (θ)$

Theorem 1.

3.1.1 Sparse linear regression

Corollary 1.

3.1.2 Group Sparse model

Corollary 2.

3.2 Non-convex $G (θ)$

Theorem 2.

3.2.1 Linear regression with SCAD regularization

Corollary 3.

3.2.2 Linear regression with noisy covariates

3.3 Corrected Lasso

Corollary 4.

3.4 Extension to Generalized linear model

4 Empirical Result

SAGA and Restricted Strong Convexity

Authors

Linear convergence of SDCA in statistical estimation

Linear Convergence of SVRG in Statistical Estimation

A Unified Convergence Analysis for Shuffling-Type Gradient Methods

Non-convex Global Minimization and False Discovery Rate Control for the TREX

Improved SVRG for Non-Strongly-Convex or Sum-of-Non-Convex Objectives

A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives

Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression

This week in AI

1 Introduction

1.1 Related work

2 Preliminaries

2.1 Restricted Strong Convexity

2.2 Assumptions for the Convex regularizer Ψ(θ)

2.2.1 Decomposibility of Ψ(θ)

2.2.2 Subspace compatibility

2.3 Assumptions for the Nonconvex regularizer gλ,μ(θ)

2.4 Implementation of the algorithm

3 Main result

3.1 Convex G(θ)

Theorem 1.

3.1.1 Sparse linear regression

Corollary 1.

3.1.2 Group Sparse model

Corollary 2.

3.2 Non-convex G(θ)

Theorem 2.

3.2.1 Linear regression with SCAD regularization

Corollary 3.

3.2.2 Linear regression with noisy covariates

3.3 Corrected Lasso

Corollary 4.

3.4 Extension to Generalized linear model

4 Empirical Result

2.2 Assumptions for the Convex regularizer $Ψ (θ)$

2.2.1 Decomposibility of $Ψ (θ)$

2.3 Assumptions for the Nonconvex regularizer $g_{λ, μ} (θ)$

3.1 Convex $G (θ)$

3.2 Non-convex $G (θ)$