Active Probabilistic Inference on Matrices for Pre-Conditioning in Stochastic Optimization

02/20/2019
by   Filip de Roos, et al.
0

Pre-conditioning is a well-known concept that can significantly improve the convergence of optimization algorithms. For noise-free problems, where good pre-conditioners are not known a priori, iterative linear algebra methods offer one way to efficiently construct them. For the stochastic optimization problems that dominate contemporary machine learning, however, this approach is not readily available. We propose an iterative algorithm inspired by classic iterative linear solvers that uses a probabilistic model to actively infer a pre-conditioner in situations where Hessian-projections can only be constructed with strong Gaussian noise. The algorithm is empirically demonstrated to efficiently construct effective pre-conditioners for stochastic gradient descent and its variants. Experiments on problems of comparably low dimensionality show improved convergence. In very high-dimensional problems, such as those encountered in deep learning, the pre-conditioner effectively becomes an automatic learning-rate adaptation scheme, which we also empirically show to work well.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/22/2021

A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization

Machine learning practitioners invest significant manual and computation...
research
08/06/2020

Iterative Pre-Conditioning for Expediting the Gradient-Descent Method: The Distributed Linear Least-Squares Problem

This paper considers the multi-agent linear least-squares problem in a s...
research
10/27/2017

SGDLibrary: A MATLAB library for stochastic gradient descent algorithms

We consider the problem of finding the minimizer of a function f: R^d →R...
research
10/22/2019

The Practicality of Stochastic Optimization in Imaging Inverse Problems

In this work we investigate the practicality of stochastic gradient desc...
research
05/22/2017

Follow the Signs for Robust Stochastic Optimization

Stochastic noise on gradients is now a common feature in machine learnin...
research
03/29/2017

Probabilistic Line Searches for Stochastic Optimization

In deterministic optimization, line searches are a standard tool ensurin...
research
09/19/2022

On the Theoretical Properties of Noise Correlation in Stochastic Optimization

Studying the properties of stochastic noise to optimize complex non-conv...

Please sign up or login with your details

Forgot password? Click here to reset