-
Training Neural Networks with Stochastic Hessian-Free Optimization
Hessian-free (HF) optimization has been successfully used for training d...
read it
-
Gradient-based Hyperparameter Optimization through Reversible Learning
Tuning hyperparameters of learning algorithms is hard because gradients ...
read it
-
BackPACK: Packing more into backprop
Automatic differentiation frameworks are optimized for exactly one thing...
read it
-
Second-order step-size tuning of SGD for non-convex optimization
In view of a direct and simple improvement of vanilla SGD, this paper pr...
read it
-
Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties
Many popular adaptive gradient methods such as Adam and RMSProp rely on ...
read it
-
Reducing the variance in online optimization by transporting past gradients
Most stochastic optimization methods use gradients once before discardin...
read it
-
Truncated Back-propagation for Bilevel Optimization
Bilevel optimization has been recently revisited for designing and analy...
read it
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters. Based on this intuition, we explore the use of exact per-sample Hessian-vector products and gradients to construct optimizers that are self-tuning and hyperparameter-free. Based on a dynamics model of the gradient, we derive a process which leads to a curvature-corrected, noise-adaptive online gradient estimate. The smoothness of our updates makes it more amenable to simple step size selection schemes, which we also base off of our estimates quantities. We prove that our model-based procedure converges in the noisy quadratic setting. Though we do not see similar gains in deep learning tasks, we can match the performance of well-tuned optimizers and ultimately, this is an interesting step for constructing self-tuning optimizers.
READ FULL TEXT
Comments
There are no comments yet.