Guided evolutionary strategies: escaping the curse of dimensionality in random search

06/26/2018
by   Niru Maheswaranathan, et al.
2

Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute than the full gradient (e.g. in meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate (e.g. in certain reinforcement learning applications, or when using synthetic gradients). We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along with random search. We define a search distribution for evolutionary strategies that is elongated along a guiding subspace spanned by the surrogate gradients. This allows us to estimate a descent direction which can then be passed to a first-order optimizer. We analytically and numerically characterize the tradeoffs that result from tuning how strongly the search distribution is stretched along the guiding subspace, and we use this to derive a setting of the hyperparameters that works well across problems. Finally, we apply our method to example problems including truncated unrolled optimization and a synthetic gradient problem, demonstrating improvement over both standard evolutionary strategies and first-order methods that directly follow the surrogate gradient. We provide a demo of Guided ES at: https://github.com/brain-research/guided-evolutionary-strategies.

READ FULL TEXT
research
10/11/2019

Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions

Evolutionary Strategies (ES) are known to be an effective black-box opti...
research
06/18/2020

An adaptive stochastic gradient-free approach for high-dimensional blackbox optimization

In this work, we propose a novel adaptive stochastic gradient-free (ASGF...
research
10/01/2021

Guiding Evolutionary Strategies by Differentiable Robot Simulators

In recent years, Evolutionary Strategies were actively explored in robot...
research
02/13/2020

Effective Reinforcement Learning through Evolutionary Surrogate-Assisted Prescription

There is now significant historical data available on decision making in...
research
06/07/2018

Stein Variational Gradient Descent Without Gradient

Stein variational gradient decent (SVGD) has been shown to be a powerful...
research
02/20/2021

Info-Evo: Using Information Geometry to Guide Evolutionary Program Learning

A novel optimization strategy, Info-Evo, is described, in which natural ...
research
03/01/2021

Fast threshold optimization for multi-label audio tagging using Surrogate gradient learning

Multi-label audio tagging consists of assigning sets of tags to audio re...

Please sign up or login with your details

Forgot password? Click here to reset