DeepAI AI Chat
Log In Sign Up

Beyond backpropagation: implicit gradients for bilevel optimization

by   Nicolas Zucchet, et al.
ETH Zurich

This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defined through a quantity that they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers and even physical systems, and allows for greater modeling flexibility compared to an explicit definition of such systems. Here we focus on gradient-based approaches that solve such problems. We distinguish them in two categories: those rooted in implicit differentiation, and those that leverage the equilibrium propagation theorem. We present the mathematical foundations that are behind such methods, introduce the gradient-estimation algorithms in detail and compare the competitive advantages of the different approaches.


page 1

page 2

page 3

page 4


Nonsmooth Implicit Differentiation for Machine Learning and Optimization

In view of training increasingly complex learning architectures, we esta...

Backpropagation for Implicit Spectral Densities

Most successful machine intelligence systems rely on gradient-based lear...

On Training Implicit Models

This paper focuses on training implicit models of infinite layers. Speci...

Agnostic Physics-Driven Deep Learning

This work establishes that a physical system can perform statistical lea...

Implicitly Defined Layers in Neural Networks

In conventional formulations of multilayer feedforward neural networks, ...

Topological Gradient-based Competitive Learning

Topological learning is a wide research area aiming at uncovering the mu...

Aerostructural Wing Shape Optimization assisted by Algorithmic Differentiation

With more efficient structures, last trends in aeronautics have witnesse...