Stable Invariant Models via Koopman Spectra

07/15/2022
by   Takuya Konishi, et al.
0

Weight-tied models have attracted attention in the modern development of neural networks. The deep equilibrium model (DEQ) represents infinitely deep neural networks with weight-tying, and recent studies have shown the potential of this type of approach. DEQs are needed to iteratively solve root-finding problems in training and are built on the assumption that the underlying dynamics determined by the models converge to a fixed point. In this paper, we present the stable invariant model (SIM), a new class of deep models that in principle approximates DEQs under stability and extends the dynamics to more general ones converging to an invariant set (not restricted in a fixed point). The key ingredient in deriving SIMs is a representation of the dynamics with the spectra of the Koopman and Perron–Frobenius operators. This perspective approximately reveals stable dynamics with DEQs and then derives two variants of SIMs. We also propose an implementation of SIMs that can be learned in the same way as feedforward models. We illustrate the empirical performance of SIMs with experiments and demonstrate that SIMs achieve comparative or superior performance against DEQs in several learning tasks.

READ FULL TEXT
research
06/28/2021

Stabilizing Equilibrium Models by Jacobian Regularization

Deep equilibrium networks (DEQs) are a new class of models that eschews ...
research
04/13/2021

Recurrent Equilibrium Networks: Unconstrained Learning of Stable and Robust Dynamical Models

This paper introduces recurrent equilibrium networks (RENs), a new class...
research
08/16/2022

On the generalization of learning algorithms that do not converge

Generalization analyses of deep learning typically assume that the train...
research
09/03/2019

Deep Equilibrium Models

We present a new approach to modeling sequential data: the deep equilibr...
research
05/27/2021

Optimization Induced Equilibrium Networks

Implicit equilibrium models, i.e., deep neural networks (DNNs) defined b...
research
10/19/2022

An Optimization-Based Supervised Learning Algorithm for PXRD Phase Fraction Estimation

In powder diffraction data analysis, phase identification is the process...
research
05/31/2023

Spontaneous symmetry breaking in generative diffusion models

Generative diffusion models have recently emerged as a leading approach ...

Please sign up or login with your details

Forgot password? Click here to reset