Exploiting Problem Structure in Deep Declarative Networks: Two Case Studies

02/24/2022
by   Stephen Gould, et al.
0

Deep declarative networks and other recent related works have shown how to differentiate the solution map of a (continuous) parametrized optimization problem, opening up the possibility of embedding mathematical optimization problems into end-to-end learnable models. These differentiability results can lead to significant memory savings by providing an expression for computing the derivative without needing to unroll the steps of the forward-pass optimization procedure during the backward pass. However, the results typically require inverting a large Hessian matrix, which is computationally expensive when implemented naively. In this work we study two applications of deep declarative networks – robust vector pooling and optimal transport – and show how problem structure can be exploited to obtain very efficient backward pass computations in terms of both time and memory. Our ideas can be used as a guide for improving the computational performance of other novel deep declarative nodes.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/26/2023

PMaF: Deep Declarative Layers for Principal Matrix Features

We explore two differentiable deep declarative layers, namely least squa...
research
09/11/2019

Deep Declarative Networks: A New Hope

We introduce a new class of end-to-end learnable models wherein data pro...
research
01/28/2020

f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation

Deep neural networks have become a mainstream approach to interactive se...
research
07/22/2022

Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning

Deep Neural Network (DNN) models are usually trained sequentially from o...
research
06/01/2021

SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models

In recent years, implicit deep learning has emerged as a method to incre...
research
12/13/2022

Regularized Optimal Transport Layers for Generalized Global Pooling Operations

Global pooling is one of the most significant operations in many machine...
research
01/09/2021

Training Deep Architectures Without End-to-End Backpropagation: A Brief Survey

This tutorial paper surveys training alternatives to end-to-end backprop...

Please sign up or login with your details

Forgot password? Click here to reset