The Natural Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its Use

05/08/2012

∙

by   Jascha Sohl-Dickstein, et al.

∙

0

∙

share

The natural gradient allows for more efficient gradient descent by removing dependencies and biases inherent in a function's parameterization. Several papers present the topic thoroughly and precisely. It remains a very difficult idea to get your head around however. The intent of this note is to provide simple intuition for the natural gradient and its use. We review how an ill conditioned parameter space can undermine learning, introduce the natural gradient by analogy to the more widely understood concept of signal whitening, and present tricks and specific prescriptions for applying the natural gradient to learning problems.