Optimizing Simulations with Noise-Tolerant Structured Exploration

by   Krzysztof Choromanski, et al.

We propose a simple drop-in noise-tolerant replacement for the standard finite difference procedure used ubiquitously in blackbox optimization. In our approach, parameter perturbation directions are defined by a family of structured orthogonal matrices. We show that at the small cost of computing a Fast Walsh-Hadamard/Fourier Transform (FWHT/FFT), such structured finite differences consistently give higher quality approximation of gradients and Jacobians in comparison to vanilla approaches that use coordinate directions or random Gaussian perturbations. We find that trajectory optimizers like Iterative LQR and Differential Dynamic Programming require fewer iterations to solve several classic continuous control tasks when our methods are used to linearize noisy, blackbox dynamics instead of standard finite differences. By embedding structured exploration in a quasi-Newton optimizer (LBFGS), we are able to learn agile walking and turning policies for quadruped locomotion, that successfully transfer from simulation to actual hardware.We theoretically justify our methods via bounds on the quality of gradient reconstruction and provide a basis for applying them also to nonsmooth problems.


page 1

page 7


Orthogonal iterations on Structured Pencils

We present a class of fast subspace tracking algorithms based on orthogo...

Structured Evolution with Compact Architectures for Scalable Policy Optimization

We present a new method of blackbox optimization via gradient approximat...

Dynamic Walking: Toward Agile and Efficient Bipedal Robots

Dynamic walking on bipedal robots has evolved from an idea in science fi...

Orthogonal Random Features

We present an intriguing discovery related to Random Fourier Features: i...

Random noise attenuation on finite-difference wave propagation using fuzzy transform

Fuzzy Transform (F-transform) has been introduced as an approximation me...

The linearization methods as a basis to derive the relaxation and the shooting methods

This chapter investigates numerical solution of nonlinear two-point boun...

ES Is More Than Just a Traditional Finite-Difference Approximator

An evolution strategy (ES) variant recently attracted significant attent...