Deep Declarative Networks: A New Hope

by   Stephen Gould, et al.

We introduce a new class of end-to-end learnable models wherein data processing nodes (or network layers) are defined in terms of desired behavior rather than an explicit forward function. Specifically, the forward function is implicitly defined as the solution to a mathematical optimization problem. Consistent with nomenclature in the programming languages community, we name our models deep declarative networks. Importantly, we show that the class of deep declarative networks subsumes current deep learning models. Moreover, invoking the implicit function theorem, we show how gradients can be back-propagated through declaratively defined data processing nodes thereby enabling end-to-end learning. We show how these declarative processing nodes can be implemented in the popular PyTorch deep learning software library allowing declarative and imperative nodes to co-exist within the same network. We provide numerous insights and illustrative examples of declarative nodes and demonstrate their application for image and point cloud classification tasks.



There are no comments yet.


page 1

page 2

page 3

page 4


Implicitly Defined Layers in Neural Networks

In conventional formulations of multilayer feedforward neural networks, ...

Optimization Induced Equilibrium Networks

Implicit equilibrium models, i.e., deep neural networks (DNNs) defined b...

cilantro: a lean, versatile, and efficient library for point cloud data processing

We introduce cilantro, an open-source C++ library for geometric and gene...

Implicit Deep Learning

We define a new class of "implicit" deep learning prediction rules that ...

Neurogenesis Deep Learning

Neural machine learning methods, such as deep neural networks (DNN), hav...

Fixed Point Networks: Implicit Depth Models with Jacobian-Free Backprop

A growing trend in deep learning replaces fixed depth models by approxim...

End-to-end Deep Prototype and Exemplar Models for Predicting Human Behavior

Traditional models of category learning in psychology focus on represent...

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.