Stochastic Multi-level Composition Optimization Algorithms with Level-Independent Convergence Rates

08/24/2020
by   Krishnakumar Balasubramanian, et al.
0

In this paper, we study smooth stochastic multi-level composition optimization problems, where the objective function is a nested composition of T functions. We assume access to noisy evaluations of the functions and their gradients, through a stochastic first-order oracle. For solving this class of problems, we propose two algorithms using moving-average stochastic estimates, and analyze their convergence to an ϵ-stationary point of the problem. We show that the first algorithm, which is a generalization of [22] to the T level case, can achieve a sample complexity of 𝒪(1/ϵ^6) by using mini-batches of samples in each iteration. By modifying this algorithm using linearized stochastic estimates of the function values, we improve the sample complexity to 𝒪(1/ϵ^4). This modification also removes the requirement of having a mini-batch of samples in each iteration. To the best of our knowledge, this is the first time that such an online algorithm designed for the (un)constrained multi-level setting, obtains the same sample complexity of the smooth single-level setting, under mild assumptions on the stochastic first-order oracle.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

08/29/2019

Multi-Level Composite Stochastic Optimization via Nested Variance Reduction

We consider multi-level composite optimization problems where each mappi...
05/05/2021

Randomized Stochastic Variance-Reduced Methods for Stochastic Bilevel Optimization

In this paper, we consider non-convex stochastic bilevel optimization (S...
05/19/2021

Trilevel and Multilevel Optimization using Monotone Operator Theory

We consider rather a general class of multi-level optimization problems,...
11/12/2019

Nonconvex Stochastic Nested Optimization via Stochastic ADMM

We consider the stochastic nested composition optimization problem where...
12/17/2020

Stochastic Compositional Gradient Descent under Compositional constraints

This work studies constrained stochastic optimization problems where the...
10/22/2021

Projection-Free Algorithm for Stochastic Bi-level Optimization

This work presents the first projection-free algorithm to solve stochast...
02/17/2020

Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization

We develop two new stochastic Gauss-Newton algorithms for solving a clas...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.