SEBOOST - Boosting Stochastic Learning Using Subspace Optimization Techniques

by   Elad Richardson, et al.

We present SEBOOST, a technique for boosting the performance of existing stochastic optimization methods. SEBOOST applies a secondary optimization process in the subspace spanned by the last steps and descent directions. The method was inspired by the SESOP optimization method for large-scale problems, and has been adapted for the stochastic learning framework. It can be applied on top of any existing optimization method with no need to tweak the internal algorithm. We show that the method is able to boost the performance of different algorithms, and make them more robust to changes in their hyper-parameters. As the boosting steps of SEBOOST are applied between large sets of descent steps, the additional subspace optimization hardly increases the overall computational burden. We introduce two hyper-parameters that control the balance between the baseline method and the secondary optimization process. The method was evaluated on several deep learning tasks, demonstrating promising results.


AdaSmooth: An Adaptive Learning Rate Method based on Effective Ratio

It is well known that we need to choose the hyper-parameters in Momentum...

SVGD: A Virtual Gradients Descent Method for Stochastic Optimization

Inspired by dynamic programming, we propose Stochastic Virtual Gradient ...

Meta Subspace Optimization

Subspace optimization methods have the attractive property of reducing l...

Adaptive Stochastic Optimization

Optimization lies at the heart of machine learning and signal processing...

Microscopic Advances with Large-Scale Learning: Stochastic Optimization for Cryo-EM

Determining the 3D structures of biological molecules is a key problem f...

Boosting as Frank-Wolfe

Some boosting algorithms, such as LPBoost, ERLPBoost, and C-ERLPBoost, a...

Please sign up or login with your details

Forgot password? Click here to reset