Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies

12/27/2021
by   Paul Vicol, et al.
9

Unrolled computation graphs arise in many scenarios, including training RNNs, tuning hyperparameters through unrolled optimization, and training learned optimizers. Current approaches to optimizing parameters in such computation graphs suffer from high variance gradients, bias, slow updates, or large memory usage. We introduce a method called Persistent Evolution Strategies (PES), which divides the computation graph into a series of truncated unrolls, and performs an evolution strategies-based update step after each unroll. PES eliminates bias from these truncations by accumulating correction terms over the entire sequence of unrolls. PES allows for rapid parameter updates, has low memory usage, is unbiased, and has reasonable variance characteristics. We experimentally demonstrate the advantages of PES compared to several other methods for gradient estimation on synthetic tasks, and show its applicability to training learned optimizers and tuning hyperparameters.

READ FULL TEXT

page 8

page 9

page 16

page 17

page 18

page 37

page 38

research
04/21/2023

Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single

We propose an evolution strategies-based algorithm for estimating gradie...
research
10/20/2021

AdamD: Improved bias-correction in Adam

Here I present a small update to the bias-correction term in the Adam op...
research
04/21/2023

Noise-Reuse in Online Evolution Strategies

Online evolution strategies have become an attractive alternative to aut...
research
02/12/2021

Bias-Free Scalable Gaussian Processes via Randomized Truncations

Scalable Gaussian Process methods are computationally attractive, yet in...
research
09/22/2022

An Investigation of the Bias-Variance Tradeoff in Meta-Gradients

Meta-gradients provide a general approach for optimizing the meta-parame...
research
07/11/2020

Online Parameter-Free Learning of Multiple Low Variance Tasks

We propose a method to learn a common bias vector for a growing sequence...
research
05/22/2022

Limitations of a proposed correction for slow drifts in decision criterion

Trial history biases in decision-making tasks are thought to reflect sys...

Please sign up or login with your details

Forgot password? Click here to reset