A Constant Step Stochastic Douglas-Rachford Algorithm with Application to Non Separable Regularizations

1 Introduction

Many applications in the fields of machine learning

[1] and signal processing [2] require the solution of the programming problem

min x \in X F (x) + G (x)

(1)

where $X$ is an Euclidean space, $F$ and $G$ are elements of the set $Γ_{0} (X)$ of convex, lower semi-continuous and proper functions. In these contexts, $F$ often represents a cost function and $G$ a regularization term. The Douglas-Rachford algorithm is one of the most popular approach towards solving Problem (1). Given $γ > 0$ , the algorithm is written

$y_{n + 1}$	$= {prox}_{γ F} (x_{n})$
$z_{n + 1}$	$= {prox}_{γ G} (2 y_{n + 1} - x_{n})$
$x_{n + 1}$	$= x_{n} + z_{n + 1} - y_{n + 1}$	(2)

where ${prox}_{γ F}$ denotes the proximity operator of $F$ , defined for every $x \in X$ by the equation

{prox}_{γ F} (x) = arg min y \in X \frac{1}{2} ∥ x - y ∥^{2} + γ F (y) .

Assuming that a standard qualification condition holds and that the set of solutions $arg min F + G$ of (1) is not empty, the sequence $(y_{n})_{n}$ converges to an element in $arg min F + G$ as $n \to + \infty$ ([3, 4]).

In this paper, we study the case where $F$ and $G$ are integral functionals of the form

F (x) = E_{ξ} (f (x, ξ)), G (x) = E_{ξ} (g (x, ξ))

where $ξ$

is a random variable (r.v) from some probability space

(Ω, F, P)

into a measurable space

(Ξ, G)

, with distribution

μ

, and where

{f (\cdot, s), s \in Ξ}

and

{g (\cdot, s), s \in Ξ}

are subsets of

Γ_{0} (X)

. In this context, the stochastic Douglas Rachford algorithm aims to solve Problem (1) by iterating

$y_{n + 1}$	$= {prox}_{γ f (\cdot, ξ_{n + 1})} (x_{n})$
$z_{n + 1}$	$= {prox}_{γ g (\cdot, ξ_{n + 1})} (2 y_{n + 1} - x_{n})$
$x_{n + 1}$	$= x_{n} + z_{n + 1} - y_{n + 1},$	(3)

where $(ξ_{n})_{n}$ is a sequence of i.i.d copies of the random variable $ξ$ and $γ > 0$ is the constant step size. Compared to the "deterministic" Douglas Rachford algorithm (1), the stochastic Douglas Rachford algorithm (1) is an online method. The constant step size used make it implementable in adaptive signal processing or online machine learning contexts. In this algorithm, the function $F$ (resp. $G$ ) is replaced at each iteration $n$ by a random realization $f (\cdot, ξ_{n})$ (resp. $g (\cdot, ξ_{n})$ ). It can be implemented in the case where $F$ (resp. $G$ ) cannot be computed in its closed form [5, 6] or in the case where the computation of its proximity operator is demanding [7]. Compared to other online optimization algorithm like the stochastic subgradient algorithm, the algorithm (1) benefits from the numerical stability of stochastic proximal methods.

Stochastic version of the Douglas Rachford algorithm have been considered in [2, 8]. These papers consider the case where $G$ is deterministic, i.e is not written as an expectation and $F$ is written as an expectation that reduces to a sum. The latter case is also contained as a particular case of the algorithm [9]. The algorithms [10, 11] are generalizations of a partially stochastic Douglas Rachford algorithm where $G$ is deterministic. The convergence of these algorithms is obtained under a summability assumption of the noise over the iterations. The stochastic Douglas Rachford studied in this paper was implemented in an adaptive signal processing context [12] to solve a target tracking problem.

Whereas the paper [12] is mainly focused on the application to target tracking, in this work we provide theoretical basis for the algorithm (1) and convergence results. Moreover, a novel application to solve a programming problem regularized with the overlapping group lasso online is provided.

The next section introduces some notations. Section 3 is devoted to the statement of the main convergence result. In Section 4, an outline of the proof of the result in Section 3 is provided. Finally, the algorithm (1) is implemented to solve a regularized problem in Section 5.

2 Notations

For every function $g \in Γ_{0} (X)$ , $\partial g (x)$ denotes the subdifferential of $g$ at the point $x \in X$ and $\partial g_{0} (x)$ the least norm element in $\partial g (x)$ . The domain of $g$ is denoted as $dom (g)$ . It is a known fact that the closure of $dom (g)$ , denoted as $cl (dom (g))$ , is convex. For every closed convex set $C$ , we denote by $Π_{C}$ the projection operator onto $C$ . The indicator function of the set $C$ is defined by $ι_{C} (x) = 0$ if $x \in C$ , and $ι_{C} (x) = + \infty$ elsewhere. It is easy to see that $ι_{C} \in Γ_{0} (X)$ and that ${prox}_{ι_{C}} = Π_{C}$ .

The Moreau envelope of $g \in Γ_{0} (X)$ is equal to

g_{γ} (x) = min y \in X g (y) + \frac{∥ y - x ∥^{2}}{2 γ}

for every $x \in X$ . Recall that $g_{γ}$ is differentiable and $\nabla g_{γ} (x) = \frac{1}{γ} (x - {prox}_{γ g} (x)) .$ If $f \in Γ_{0} (X)$ is differentiable, then, $\partial f (x) = {\nabla f (x)}$ and $\nabla f ({prox}_{γ f} (x)) = \nabla f_{γ} (x)$ , for every $x \in X$ .

When $S \subset X$ , $d (x, S)$ denote the distance from the point $x \in X$ to the set $S$ . In the context of algorithm (1) we shall denote $D (s) = dom (g (\cdot, s))$ and $D = dom (G)$ . Denote $B (X)$ the Borel sigma field over $X$ . For every $p \geq 1$ , $L^{p} (Ξ, X)$ is the set of all r.v $φ$ from the probability space $(Ξ, G, μ)$ into the measurable space $(X, B (X))$ , such that $∥ φ ∥^{p}$ is integrable.

From now on, we shall state explicitly the dependence of the iterates of the algorithm in the step size and the starting point. Namely, we shall denote $(x_{n}^{γ, ν})_{n}$ the sequence $(x_{n})_{n}$ generated by the stochastic Douglas Rachford algorithm (1) with step $γ$ , such that the distribution of $x_{0}^{γ, ν}$ over $X$ is $ν$ . If $ν = δ_{a}$ , where $δ_{a}$ is the Dirac measure at the point $a \in X$ , we shall prefer the notation $x_{n}^{γ, a}$ .

3 Main convergence theorem

Consider the following assumptions.

Assumption 1.

For every compact set $K \subset X$ , there exists $ε > 0$ such that

sup x \in K \cap D \int ∥ \partial g_{0} (x, s) ∥^{1 + ε} μ (d s) < \infty .

Assumption 2.

For $μ$ -a.e $s \in Ξ$ , $f (\cdot, s)$ is differentiable and there exists a closed ball in $X$ such that $∥ \nabla f (x, s) ∥ \leq M (s)$ for all $x$ in this ball, where $M (s)$ is $μ$ -integrable. Moreover, for every compact set $K \subset X$ , there exists $ε > 0$ such that

sup x \in K \int ∥ \nabla f (x, s) ∥^{1 + ε} μ (d s) < \infty .

Assumption 3.

$\forall x \in X, \int d (x, D (s))^{2} μ (d s) \geq C d (x)^{2}$ .

Assumption 4.

For every compact set $K \subset X$ , there exists $ε, C, γ_{0} > 0$ such that for all $γ \in (0, γ_{0}]$ and all $x \in K$ ,

\frac{1}{γ^{1 + ε}} \int ∥ {prox}_{γ g (\cdot, s)} (x) - Π_{cl (D (s))} (x) ∥^{1 + ε} μ (d s) < C .

Assumption 5.

There exists $L > 0$ such that $\nabla f (\cdot, s)$ is $μ$ -a.e, a $L$ -Lipschitz continuous function.

Assumption 6.

There exists $x_{⋆} \in arg min F + G$ and $φ \in L^{2} (Ξ, X)$ such that $φ (s) \in \partial g (x_{⋆}, s)$ $μ$ -a.s, $\nabla f (x_{⋆}, \cdot) \in L^{2} (Ξ, X)$ and $\int \nabla f (x_{⋆}, s) μ (d s) + \int φ (s) μ (d s) = 0$ .

Assumption 7.

The function $F + G$ satisfies one of the following properties:

[label=()]
$F + G$ is coercive i.e $F (x) + G (x) ⟶_{∥ x ∥ \to + \infty} + \infty$
$F + G$ is supercoercive i.e $\frac{F (x) + G (x)}{∥ x ∥} ⟶_{∥ x ∥ \to + \infty} + \infty$ .

Assumption 8.

There exists $γ_{0} > 0$ , such that for all $γ \in (0, γ_{0}]$ and all $x \in X$ ,

	$\int ∥ \nabla f_{γ} (x, s) ∥ + \frac{1}{γ} ∥ {p r o x}_{γ g (\cdot, s)} (x) - Π_{cl (D (s))} (x) ∥ μ (d s)$
	$\leq C (1 + \| F^{γ} (x) + G^{γ} (x) \|) .$

Theorem 1.

Let Assumptions 1– 8 hold true. Then, for each probability measure $ν$ over $X$

having a finite second moment, for any

ε > 0

limsup n \to \infty \frac{1}{n + 1} n \sum k = 0 P (d (x_{k}^{γ, ν}, arg min (F + G)) > ε) - - \to γ \to 0 0 .

Moreover, if Assumption 7–2 holds true, then

	$limsup n \to \infty P (d ({¯ x}_{n}^{γ, ν}, arg min (F + G)) \geq ε) - - \to γ \to 0 0, and$
	$limsup n \to \infty d (E ({¯ x}_{n}^{γ, ν}), arg min (F + G)) - - \to γ \to 0 0 .$

where ${¯ x}_{n}^{γ, ν} = \frac{1}{n} \sum_{k = 1}^{n} x_{k}^{γ, ν}$ .

Loosely speaking, the theorem states that, with high probability, the iterates $(x_{n}^{γ, ν})_{n}$ stay close to the set of solutions $arg min F + G$ as $n \to \infty$ and $γ \to 0$ .

Some Assumptions deserve comments.

Following [13], we say that a finite collection of subsets $C1,\mathellipsis,Cm$ of $X$ is linearly regular if

∃κ>0,∀x∈X,maxs∈{1,\mathellipsis,m}d(x,Cs)≥κd(x,∩ms=1Cs)

In the case where there exists a $μ$ -probability one set $~ Ξ$ such that the set ${D(s),s∈~Ξ}={C1,\mathellipsis,Cm}$ is finite, it is routine to check that Assumption 3 holds if and only if the domains $C1,\mathellipsis,Cm$ are linearly regular. See [12] for an applicative context of the algorithm (1) in the latter case.

It is a known fact that

{prox}_{γ g (\cdot, s)} (x) ⟶_{γ \to 0} Π_{cl (dom (g (\cdot, s)))} (x),

for each $(x, s)$ . Assumptions 4 and 8 add controls on the convergence rate.

Since $f (\cdot, s), g (\cdot, s) \in Γ_{0} (X)$ , and $f (\cdot, s)$ is differentiable, $\partial (F + G) (x) = \nabla F (x) + \partial G (x) = E (\nabla f (x, ξ)) + E (\partial g (x, ξ))$ [14], where the set $E (\partial g (x, ξ))$ is defined by its Aumann integral

{\int φ (s) μ (d s), φ \in L^{1} (Ξ, X), s.t. % φ (s) \in \partial g (x, s), μ -a.s.}

Therefore, using Fermat’s rule, if $x \in arg min F + G$ , then there exists $φ \in L^{1} (Ξ, X)$ , such that $φ (s) \in \partial g (x, s)$ $μ$ -a.s, and $\int \nabla f (x, s) μ (d s) + \int φ (s) μ (d s) = 0$ . We refer to $(\nabla f (x, \cdot), φ)$ as a representation of the solution $x$ . Assumption 6 ensures the existence of $x_{⋆} \in arg min F + G$ with a representation $\nabla f (x, \cdot), φ \in L^{2} (Ξ, X)$ .

4 Outline of the convergence proof

This section is devoted to sketching the proof of the convergence of the stochastic Douglas Rachford algorithm. The approach follows the same steps as [6] and is detailed in [15]. The first step of the proof is to study the dynamical behavior of the iterates $(x_{n}^{γ, a})_{n}$ where $a \in D$

. The Ordinary Differential Equation (ODE) method, well known in the literature of stochastic approximation (

[16]), is applied. Consider the continuous time stochastic process

x_{γ, a}

obtained by linearly interpolating with time interval

γ

the iterates

(x_{n}^{γ, a})

x_{γ, a} (t) = x_{n}^{γ, a} + (t - n γ) \frac{x_{n + 1}^{γ, a} - x_{n}^{γ, a}}{γ},

(4)

for all $t \geq 0$ such that $n γ \leq t < (n + 1) γ$ , for all $n \in N$ . Let Assumptions 1–4 ¹¹1In the case where the domains are common, i.e $s \mapsto D (s)$ is $μ$ -a.s constant, the moment Assumptions 1 and 2 are sufficient to state the dynamical behavior result. See [12] for an applicative context where the domains $D (s)$ are distinct. hold true. Consider the set $C (R_{+}, X)$ of continuous functions from $R_{+}$ to $X$ equipped with the topology of uniform convergence on the compact intervals. It is shown that the continuous time stochastic process $x_{γ, a}$ converges weakly over $R_{+}$ (i.e in distribution in $C (R_{+}, X)$ ) as $γ \to 0$ . Moreover, the limit is proven to be the unique absolutely continuous function $x$ over $R_{+}$ satisfying $x (0) = a$ and for almost every $t \geq 0$ , the Differential Inclusion (DI),

˙ x (t) \in - (\nabla F + \partial G) (x (t)),

(5)

(see [17]). Differential inclusions like (5) generalize ODE to set-valued mappings. The DI (5) induces a map $Φ : D \times R_{+} \to D, (x_{0}, t) \mapsto x (t)$ that can be extended to a semi-flow over $cl (D)$ , still denoted by $Φ$ .

The weak convergence of $(x_{γ, a})$ to $x$ is not enough to study the long term behavior of the iterates $(x_{n}^{γ, a})_{n}$

. The second step of the proof is to prove a stability result for the Feller Markov chain

(x_{n}^{γ, a})_{n}

. Denote by

P_{γ}

its transition kernel. The deterministic counterpart of this step of the proof is the so-called Fejér monotonicity of the sequence

(x_{n})

of the algorithm (1). Even if some work has been done [5, 18], there is no immediate way to adapt the Fejér monotonicity to our random setting, mainly because of the constant step

γ

. As an alternative, we assume Hypotheses 5-6, and prove the existence of positive numbers

α, C

and

γ_{0}

, such that for every

γ \in (0, γ_{0}]

	$E_{n} ∥ x_{n + 1}^{γ, a} - x_{⋆} ∥^{2} \leq$	$∥ x_{n}^{γ, a} - x_{⋆} ∥^{2}$		(6)
		$- α γ (F^{γ} + G^{γ}) (x_{n}^{γ, a}) + γ C .$

In this inequality, $E_{n}$ denotes the conditional expectation with respect to the sigma-algebra $σ(xγ0,xγ1,\mathellipsis,xγn)$ and

F^{γ} (x) = \int f_{γ} (x, s) μ (d s), G^{γ} (x) = \int g_{γ} (x, s) μ (d s) .

Since $γ \mapsto F^{γ} (x) + G^{γ} (x)$ is decreasing [6, 15], the function $F^{γ} + G^{γ}$ can be replaced by $F^{γ_{0}} + G^{γ_{0}}$ . Besides, the coercivity of $F + G$ (Assumption 7) implies the coercivity of $F^{γ_{0}} + G^{γ_{0}}$ ( [6, 15]). Therefore, assuming 5–7 and setting $Ψ = F^{γ_{0}} + G^{γ_{0}}$ , there exist positive numbers $α, C$ and $γ_{0}$ , such that for every $γ \in (0, γ_{0}]$ ,

E_{n} ∥ x_{n + 1}^{γ, a} - x_{⋆} ∥^{2} \leq ∥ x_{n}^{γ, a} - x_{⋆} ∥^{2} - α γ Ψ (x_{n}^{γ, a}) + γ C .

(7)

Equation (7) can alternatively be seen as a tightness result. It implies that the set $I_{γ}$ of invariant measures of the Markov kernel $P_{γ}$ is not empty for every $γ \in (0, γ_{0}]$ , and that the set

Inv = \cup_{γ \in (0, γ_{0}]} I_{γ}

(8)

is tight( [19, 20]).

It remains to characterize the cluster points of Inv as $γ \to 0$ . To that end, the dynamical behavior result and the stability result are combined. Let Assumptions 1– 8 hold true. ²²2Assumptions 3, 4 and 8 are not needed if the domains $D (s)$ are common. Then, the set Inv is tight, and, as $γ \to 0$ , every cluster point of Inv is an invariant measure for the semi-flow $Φ$ . The Theorem 1 is a consequence of this fact.

5 Application to structured regularization

In this section is provided an application of the stochastic Douglas Rachford (1) algorithm to solve a regularized optimization problem. Consider problem (1), where $F$ is a cost function that is written as an expectation, and $G$ is a regularization term. Towards solving (1), many approaches involve the computation of the proximity operator of the regularization term $G$ . In the case where $G$ is a structured regularization term, its proximity operator is often difficult to compute. When $G$ is a graph-based regularization, it is possible to apply a stochastic proximal method to address the regularization [7]. We shall concentrate on the case where $G$ is an overlapping group regularization. In this case, the computation of the proximity operator of $G$ is known to be a bottleneck [21]. We shall apply the algorithm (1) to overcome this difficulty.

Consider $X = R^{N}$ , $N \in N^{⋆}$ , and $g \in N^{⋆}$ . Consider $g$ subsets of ${1,\mathellipsis,N}$ , $S1,\mathellipsis,Sg$ , possibly overlapping. Set $G (x) = \sum_{j = 1}^{g} ∥ x_{S_{j}} ∥$ , where $x_{S_{j}}$ denotes the restriction of $x$ to the set of index $S_{j}$ and $∥ \cdot ∥$ denotes the Euclidean norm. Set $F (x) = E_{(ξ, η)} (h (η ⟨ x, ξ ⟩))$ where $h$ denotes the hinge loss $h (z) = max (0, 1 - z)$ and $(ξ, η)$ is a r.v defined on some probability space with values in $X \times {- 1, + 1}$ . In this case, the problem (1) is also called the SVM classification problem, regularized by the overlapping group lasso. It is assumed that the user is provided with i.i.d copies $((ξ_{n}, η_{n}))_{n}$ of the r.v $(ξ, η)$ online.

To solve this problem, we implement a stochastic Douglas Rachford strategy. To that end, the regularization $G$ is rewritten $G (x) = E_{J} (g ∥ x_{S_{J}} ∥)$ where $J$ is an uniform r.v over ${1,\mathellipsis,g}$ . At each iteration $n$ of the stochastic Douglas Rachford algorithm, the user is provided with the realization $(ξ_{n}, η_{n})$ and sample a group $J_{n}$ uniformly in ${1,\mathellipsis,g}$ . Then, a Douglas Rachford step is done, involving the computation of the proximity operators of the functions $g_{n} : x \mapsto ∥ x_{S_{J_{n}}} ∥$ and $f_{n} : x \mapsto h (η_{n} ⟨ x, ξ_{n} ⟩)$ .

This strategy is compared with a partially stochastic Douglas Rachford algorithm, deterministic in the regularization $G$ , where the fast subroutine Fog-Lasso [21] is used to compute the proximity operator of the regularization $G$ . At each iteration $n$ , the user is provided with $(ξ_{n}, η_{n})$ . Then, a Douglas Rachford step is done, involving the computation of the proximity operators of the functions $G$ and $f_{n} : x \mapsto h (η_{n} ⟨ x, ξ_{n} ⟩)$ . Figure 1 demonstrates the advantage of treating the regularization term in a stochastic way.

Figure 1: The objective function $F + G$ as a function of time in seconds for each algorithm

Figure 2: Histogram of the Initialization and the last iterates of the Stochastic D-R (S. D-R) and the partially stochastic D-R (Part. S. D-R)

In Figure 1 "Stochastic D-R" denotes the stochastic Douglas Rachford algorithm and "Partially stochastic D-R" denotes the partially stochastic Douglas Rachford where the subroutine FoG-Lasso [21] is used at each iteration to compute the true proximity operator of the regularization $G$ . Figure 2 shows the appearance of the first and the last iterates. Even if a best performing procedure [21] is used to compute ${prox}_{γ G}$ , we observe on Figure 1 that Stochastic D-R takes advantage of being a stochastic method. This advantage is known to be twofold ([22]). First, the iteration complexity of Stochastic D-R is moderate because ${prox}_{γ G}$ is never computed. Then, Stochastic D-R is faster than its partially deterministic counterpart which uses Fog-Lasso [21] as a subroutine, especially in the first iterations of the algorithms. Moreover, Stochastic D-R seems to perform globally better. This is because every proximity operators in Stochastic D-R can be efficiently computed ([23]). Contrary to the proximity operator of $G$ [21], the proximity operator of $g_{n}$ is easily computable. The proximity operator of $f_{n}$ is easily computable as well.³³3Even if $h (x) = log (1 + exp (- x))$

(logistic regression), the proximity operator of

f_{n}

is easily computable, see [2].

Références

[1] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning, vol. 3, no. 1, pp. 1–122, 2011.
[2] G. Chierchia, A. Cherni, E. Chouzenoux, and J.-C. Pesquet, “Approche de douglas-rachford aléatoire par blocs appliquée à la régression logistique parcimonieuse,” in GRETSI, 2017.
[3] P.-L. Lions and B. Mercier, “Splitting algorithms for the sum of two nonlinear operators,” SIAM J. Numer. Anal., vol. 16, no. 6, pp. 964–979, 1979.
[4] J. Eckstein and D. P. Bertsekas, “On the douglas—rachford splitting method and the proximal point algorithm for maximal monotone operators,” Mathematical Programming, vol. 55, no. 1, pp. 293–318, 1992.
[5] P. Bianchi and W. Hachem, “Dynamical behavior of a stochastic Forward-Backward algorithm using random monotone operators,” ArXiv e-prints, 1508.02845, Aug. 2015, To be published in J. Optim. Theory Appl.
[6] P. Bianchi, W. Hachem, and A. Salim, “A constant step Forward-Backward algorithm involving random maximal monotone operators,” arXiv preprint arXiv:1702.04144, 2017.
[7] A. Salim, P. Bianchi, and W. Hachem, “Snake: a stochastic proximal gradient algorithm for regularized problems over large graphs,” in preparation, 2017.
[8] Z. Shi and R. Liu, “Online and stochastic douglas-rachford splitting method for large scale machine learning,” arXiv preprint arXiv:1308.4757, 2013.
[9] A. Chambolle, M. J Ehrhardt, P. Richtárik, and C.-B. Schönlieb, “Stochastic primal-dual hybrid gradient algorithm with arbitrary sampling and imaging application,” arXiv preprint arXiv:1706.04957, 2017.
[10] L. Rosasco, S. Villa, and B.-C. Vu, “Stochastic inertial primal-dual algorithms,” arXiv preprint arXiv:1507.00852, 2015.
[11] P.-.L Combettes and J.-C. Pesquet, “Stochastic forward-backward and primal-dual approximation algorithms with application to online image restoration,” in Signal Processing Conference (EUSIPCO), 2016 24th European. IEEE, 2016, pp. 1813–1817.
[12] R. Mourya, P. Bianchi, A. Salim, and C. Richard, “An adaptive distributed asynchronous algorithm with application to target localization,” in CAMSAP, 2017.
[13] H. H. Bauschke and J. M. Borwein, “On projection algorithms for solving convex feasibility problems,” SIAM review, vol. 38, no. 3, pp. 367–426, 1996.
[14] R. T. Rockafellar and R. J.-B. Wets, Variational analysis, vol. 317 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], Springer-Verlag, Berlin, 1998.
[15] A. Salim, P. Bianchi, and W. Hachem, “A stochastic douglas-rachford algorithm with constant step size,” Tech. Rep., see https://adil-salim.github.io/Research, 2017.
[16] H. J. Kushner and G. G. Yin, Stochastic approximation and recursive algorithms and applications, vol. 35 of Applications of Mathematics (New York), Springer-Verlag, New York, second edition, 2003, Stochastic Modelling and Applied Probability.
[17] H. Brézis, Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert, North-Holland mathematics studies. Elsevier Science, Burlington, MA, 1973.
[18] P.-L. Combettes and J.-C. Pesquet, “Stochastic quasi-fejér block-coordinate fixed point iterations with random sweeping,” SIAM Journal on Optimization, vol. 25, no. 2, pp. 1221–1248, 2015.
[19] J.-C. Fort and G. Pagès, “Asymptotic behavior of a Markovian stochastic algorithm with constant step,” SIAM J. Control Optim., vol. 37, no. 5, pp. 1456–1482 (electronic), 1999.
[20] P. Bianchi, W. Hachem, and A. Salim, “Asymptotics of constant step stochastic approximations involving differential inclusions,” arXiv preprint arXiv:1612.03831, 2016.
[21] L. Yuan, J. Liu, and J. Ye, “Efficient methods for overlapping group lasso,” in Advances in NIPS, 2011, pp. 352–360.
[22] L. Bottou, F. E Curtis, and J. Nocedal, “Optimization methods for large-scale machine learning,” arXiv preprint arXiv:1606.04838, 2016.
[23] H. H. Bauschke and P. L. Combettes, Convex analysis and monotone operator theory in Hilbert spaces, CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC. Springer, New York, 2011.

A Constant Step Stochastic Douglas-Rachford Algorithm with Application to Non Separable Regularizations

Authors

Linear Stochastic Approximation: Constant Step-Size and Iterate Averaging

Completely Uncoupled Algorithms for Network Utility Maximization

Chirotopes of Random Points in Space are Realizable on a Small Integer Grid

Fixed Point Analysis of Douglas-Rachford Splitting for Ptychography and Phase Retrieval

Convergence of constant step stochastic gradient descent for non-smooth non-convex functions

Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

Margins, Shrinkage, and Boosting

1 Introduction

2 Notations

3 Main convergence theorem

Assumption 1.

Assumption 2.

Assumption 3.

Assumption 4.

Assumption 5.

Assumption 6.

Assumption 7.

Assumption 8.

Theorem 1.

4 Outline of the convergence proof

5 Application to structured regularization

Références

A Constant Step Stochastic Douglas-Rachford Algorithm with Application to Non Separable Regularizations

Authors

Related Research

Linear Stochastic Approximation: Constant Step-Size and Iterate Averaging

Completely Uncoupled Algorithms for Network Utility Maximization

Chirotopes of Random Points in Space are Realizable on a Small Integer Grid

Fixed Point Analysis of Douglas-Rachford Splitting for Ptychography and Phase Retrieval

Convergence of constant step stochastic gradient descent for non-smooth non-convex functions

Non-ergodic Convergence Analysis of Heavy-Ball Algorithms

Margins, Shrinkage, and Boosting

This week in AI

1 Introduction

2 Notations

3 Main convergence theorem

Assumption 1.

Assumption 2.

Assumption 3.

Assumption 4.

Assumption 5.

Assumption 6.

Assumption 7.

Assumption 8.

Theorem 1.

4 Outline of the convergence proof

5 Application to structured regularization

Références