One-step differentiation of iterative algorithms

05/23/2023
by   Jérôme Bolte, et al.
0

In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as performant as implicit differentiation for fast algorithms (e.g., superlinear optimization methods). We provide a complete theoretical approximation analysis with specific examples (Newton's method, gradient descent) along with its consequences in bilevel optimization. Several numerical examples illustrate the well-foundness of the one-step estimator.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/28/2021

Efficient Automatic Differentiation of Implicit Functions

Derivative-based algorithms are ubiquitous in statistics, machine learni...
research
05/05/2023

Software-based Automatic Differentiation is Flawed

Various software efforts embrace the idea that object oriented programmi...
research
02/10/2020

Super-efficiency of automatic differentiation for functions defined as a minimum

In min-min optimization or max-min optimization, one has to compute the ...
research
11/23/2021

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation

Most set prediction models in deep learning use set-equivariant operatio...
research
07/02/2022

Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation

Iterative refinement – start with a random guess, then iteratively impro...
research
11/29/2021

Amortized Implicit Differentiation for Stochastic Bilevel Optimization

We study a class of algorithms for solving bilevel optimization problems...
research
05/13/2022

A Unified Framework for Implicit Sinkhorn Differentiation

The Sinkhorn operator has recently experienced a surge of popularity in ...

Please sign up or login with your details

Forgot password? Click here to reset