Stochastic Gauss-Newton Algorithms for Nonconvex Compositional Optimization

02/17/2020
by   Quoc Tran-Dinh, et al.
0

We develop two new stochastic Gauss-Newton algorithms for solving a class of stochastic nonconvex compositional optimization problems frequently arising in practice. We consider both the expectation and finite-sum settings under standard assumptions. We use both classical stochastic and SARAH estimators for approximating function values and Jacobians. In the expectation case, we establish O(ε^-2) iteration complexity to achieve a stationary point in expectation and estimate the total number of stochastic oracle calls for both function values and its Jacobian, where ε is a desired accuracy. In the finite sum case, we also estimate the same iteration complexity and the total oracle calls with high probability. To our best knowledge, this is the first time such global stochastic oracle complexity is established for stochastic Gauss-Newton methods. We illustrate our theoretical results via numerical examples on both synthetic and real datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/05/2016

Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

In this paper we study stochastic quasi-Newton methods for nonconvex sto...
research
07/08/2019

A Hybrid Stochastic Optimization Framework for Stochastic Composite Nonconvex Optimization

In this paper, we introduce a new approach to develop stochastic optimiz...
research
02/02/2022

DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization

We develop and analyze DASHA: a new family of methods for nonconvex dist...
research
10/25/2022

An Optimal Stochastic Algorithm for Decentralized Nonconvex Finite-sum Optimization

This paper studies the synchronized decentralized nonconvex optimization...
research
06/19/2021

SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums

We present a principled approach for designing stochastic Newton methods...
research
08/29/2019

Multi-Level Composite Stochastic Optimization via Nested Variance Reduction

We consider multi-level composite optimization problems where each mappi...

Please sign up or login with your details

Forgot password? Click here to reset