State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks

09/19/2022
by   Alicia Y. Tsai, et al.
0

Implicit models are a general class of learning models that forgo the hierarchical layer structure typical in neural networks and instead define the internal states based on an “equilibrium” equation, offering competitive performance and reduced memory consumption. However, training such models usually relies on expensive implicit differentiation for backward propagation. In this work, we present a new approach to training implicit models, called State-driven Implicit Modeling (SIM), where we constrain the internal states and outputs to match that of a baseline model, circumventing costly backward computations. The training problem becomes convex by construction and can be solved in a parallel fashion, thanks to its decomposable structure. We demonstrate how the SIM approach can be applied to significantly improve sparsity (parameter reduction) and robustness of baseline models trained on FashionMNIST and CIFAR-100 datasets.

READ FULL TEXT
research
04/23/2023

Efficient Training of Deep Equilibrium Models

Deep equilibrium models (DEQs) have proven to be very powerful for learn...
research
11/09/2021

On Training Implicit Models

This paper focuses on training implicit models of infinite layers. Speci...
research
09/29/2021

Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State

Spiking neural networks (SNNs) are brain-inspired models that enable ene...
research
09/03/2019

Deep Equilibrium Models

We present a new approach to modeling sequential data: the deep equilibr...
research
12/10/2021

Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach

Implicit neural networks are a general class of learning models that rep...
research
08/17/2019

Implicit Deep Learning

We define a new class of "implicit" deep learning prediction rules that ...
research
07/16/2023

Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks

Implicit models such as Deep Equilibrium Models (DEQs) have garnered sig...

Please sign up or login with your details

Forgot password? Click here to reset