
BarzilaiBorwein Step Size for Stochastic Gradient Descent
One of the major issues in stochastic gradient descent (SGD) methods is ...
read it

Balancing Rates and Variance via Adaptive BatchSize for Stochastic Optimization Problems
Stochastic gradient descent is a canonical tool for addressing stochasti...
read it

Riemannian stochastic variance reduced gradient
Stochastic variance reduction algorithms have recently become popular fo...
read it

Constant Step Size LeastMeanSquare: BiasVariance Tradeoffs and Optimal Sampling Distributions
We consider the leastsquares regression problem and provide a detailed ...
read it

Iterative exact global histogram specification and SSIM gradient ascent: a proof of convergence, step size and parameter selection
The SSIMoptimized exact global histogram specification (EGHS) is shown ...
read it

Minimizing Quantum Renyi Divergences via Mirror Descent with Polyak Step Size
Quantum information quantities play a substantial role in characterizing...
read it

On Stochastic Variance Reduced Gradient Method for Semidefinite Optimization
The lowrank stochastic semidefinite optimization has attracted rising a...
read it
On stochastic mirror descent with interacting particles: convergence properties and variance reduction
An open problem in optimization with noisy information is the computation of an exact minimizer that is independent of the amount of noise. A standard practice in stochastic approximation algorithms is to use a decreasing stepsize. However, to converge the stepsize must decrease exponentially slow, and therefore this approach is not useful in practice. A second alternative is to use a fixed stepsize and run independent replicas of the algorithm and average these. A third option is to run replicas of the algorithm and allow them to interact. It is unclear which of these options works best. To address this question, we reduce the problem of the computation of an exact minimizer with noisy gradient information to the study of stochastic mirror descent with interacting particles. We study the convergence of stochastic mirror descent and make explicit the tradeoffs between communication and variance reduction. We provide theoretical and numerical evidence to suggest that interaction helps to improve convergence and reduce the variance of the estimate.
READ FULL TEXT
Comments
There are no comments yet.