Learning with Algorithmic Supervision via Continuous Relaxations

10/11/2021
by   Felix Petersen, et al.
0

The integration of algorithmic components into neural architectures has gained increased attention recently, as it allows training neural networks with new forms of supervision such as ordering constraints or silhouettes instead of using ground truth labels. Many approaches in the field focus on the continuous relaxation of a specific task and show promising results in this context. But the focus on single tasks also limits the applicability of the proposed concepts to a narrow range of applications. In this work, we build on those ideas to propose an approach that allows to integrate algorithms into end-to-end trainable neural network architectures based on a general approximation of discrete conditions. To this end, we relax these conditions in control structures such as conditional statements, loops, and indexing, so that resulting algorithms are smoothly differentiable. To obtain meaningful gradients, each relevant variable is perturbed via logistic distributions and the expectation value under this perturbation is approximated. We evaluate the proposed continuous relaxation model on four challenging tasks and show that it can keep up with relaxations specifically designed for each individual task.

READ FULL TEXT

page 7

page 8

page 9

research
05/09/2021

Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

Sorting and ranking supervision is a method for training neural networks...
research
09/01/2022

Learning with Differentiable Algorithms

Classic algorithms and machine learning systems like neural networks are...
research
06/05/2023

End-to-end Differentiable Clustering with Associative Memories

Clustering is a widely used unsupervised learning technique involving an...
research
07/26/2018

Superpixel Sampling Networks

Superpixels provide an efficient low/mid-level representation of image d...
research
04/26/2017

Limits of End-to-End Learning

End-to-end learning refers to training a possibly complex learning syste...
research
06/14/2019

Augmenting Neural Networks with First-order Logic

Today, the dominant paradigm for training neural networks involves minim...

Please sign up or login with your details

Forgot password? Click here to reset