DeepAI AI Chat
Log In Sign Up

Beyond backpropagation: implicit gradients for bilevel optimization

05/06/2022
by   Nicolas Zucchet, et al.
ETH Zurich
0

This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defined through a quantity that they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers and even physical systems, and allows for greater modeling flexibility compared to an explicit definition of such systems. Here we focus on gradient-based approaches that solve such problems. We distinguish them in two categories: those rooted in implicit differentiation, and those that leverage the equilibrium propagation theorem. We present the mathematical foundations that are behind such methods, introduce the gradient-estimation algorithms in detail and compare the competitive advantages of the different approaches.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/08/2021

Nonsmooth Implicit Differentiation for Machine Learning and Optimization

In view of training increasingly complex learning architectures, we esta...
06/01/2018

Backpropagation for Implicit Spectral Densities

Most successful machine intelligence systems rely on gradient-based lear...
11/09/2021

On Training Implicit Models

This paper focuses on training implicit models of infinite layers. Speci...
05/30/2022

Agnostic Physics-Driven Deep Learning

This work establishes that a physical system can perform statistical lea...
03/03/2020

Implicitly Defined Layers in Neural Networks

In conventional formulations of multilayer feedforward neural networks, ...
08/21/2020

Topological Gradient-based Competitive Learning

Topological learning is a wide research area aiming at uncovering the mu...
09/26/2020

Aerostructural Wing Shape Optimization assisted by Algorithmic Differentiation

With more efficient structures, last trends in aeronautics have witnesse...