Monotone operator equilibrium networks

06/15/2020
by   Ezra Winston, et al.
0

Implicit-depth models such as Deep Equilibrium Networks have recently been shown to match or exceed the performance of traditional deep networks while being much more memory efficient. However, these models suffer from unstable convergence to a solution and lack guarantees that a solution exists. On the other hand, Neural ODEs, another class of implicit-depth models, do guarantee existence of a unique solution but perform poorly compared with traditional networks. In this paper, we develop a new class of implicit-depth model based on the theory of monotone operators, the Monotone Operator Equilibrium Network (MON). We show the close connection between finding the equilibrium point of an implicit network and solving a form of monotone operator splitting problem, which admits efficient solvers with guaranteed, stable convergence. We then develop a parameterization of the network which ensures that all operators remain monotone, which guarantees the existence of a unique equilibrium point. Finally, we show how to instantiate several versions of these models, and implement the resulting iterative solvers, for structured linear operators such as multi-scale convolutions. The resulting models vastly outperform the Neural ODE-based models while also being more computationally efficient. Code is available at http://github.com/locuslab/monotone_op_net.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/28/2021

Stabilizing Equilibrium Models by Jacobian Regularization

Deep equilibrium networks (DEQs) are a new class of models that eschews ...
research
10/14/2021

Effective Certification of Monotone Deep Equilibrium Models

Monotone Operator Equilibrium Models (monDEQs) represent a class of mode...
research
05/27/2021

Optimization Induced Equilibrium Networks

Implicit equilibrium models, i.e., deep neural networks (DNNs) defined b...
research
11/22/2021

Improved Model based Deep Learning using Monotone Operator Learning (MOL)

Model-based deep learning (MoDL) algorithms that rely on unrolling are e...
research
12/10/2021

Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach

Implicit neural networks are a general class of learning models that rep...
research
05/27/2022

Global Convergence of Over-parameterized Deep Equilibrium Models

A deep equilibrium model (DEQ) is implicitly defined through an equilibr...
research
06/06/2022

Stable and memory-efficient image recovery using monotone operator learning (MOL)

We introduce a monotone deep equilibrium learning framework for large-sc...

Please sign up or login with your details

Forgot password? Click here to reset