Gradients are Not All You Need

11/10/2021
by   Luke Metz, et al.
14

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/23/2022

Augmenting differentiable physics with randomized smoothing

In the past few years, following the differentiable programming paradigm...
research
12/31/2020

Differentiable Programming à la Moreau

The notion of a Moreau envelope is central to the analysis of first-orde...
research
04/28/2023

Improving Gradient Computation for Differentiable Physics Simulation with Contacts

Differentiable simulation enables gradients to be back-propagated throug...
research
07/17/2019

Zygote: A Differentiable Programming System to Bridge Machine Learning and Scientific Computing

Scientific computing is increasingly incorporating the advancements in m...
research
07/17/2019

A Differentiable Programming System to Bridge Machine Learning and Scientific Computing

Scientific computing is increasingly incorporating the advancements in m...
research
02/11/2021

Differentiable Implicit Soft-Body Physics

We present a differentiable soft-body physics simulator that can be comp...
research
03/29/2016

Dataflow Matrix Machines as a Generalization of Recurrent Neural Networks

Dataflow matrix machines are a powerful generalization of recurrent neur...

Please sign up or login with your details

Forgot password? Click here to reset