Differentiable Approximation Bridges For Training Networks Containing Non-Differentiable Functions

05/09/2019
by   Jason Ramapuram, et al.
0

Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropation for efficient calculation of gradients. In this work, we introduce a novel method to allow for non-differentiable functions at intermediary layers of deep neural networks. We do so through the introduction of a differentiable approximation bridge (DAB) neural network which provides smooth approximations to the gradient of the non-differentiable function. We present strong empirical results (performing over 600 experiments) in three different domains: unsupervised (image) representation learning, image classification, and sequence sorting to demonstrate that our proposed method improves state of the art performance. We demonstrate that utilizing non-differentiable functions in unsupervised (image) representation learning improves reconstruction quality and posterior linear separability by 10 also observe an accuracy improvement of 77 25 classification setting with the sort non-linearity. This work enables the usage of functions that were previously not usable in neural networks.

READ FULL TEXT
research
11/03/2020

Analytical aspects of non-differentiable neural networks

Research in computational deep learning has directed considerable effort...
research
07/16/2020

Interpretable Neuroevolutionary Models for Learning Non-Differentiable Functions and Programs

A key factor in the modern success of deep learning is the astonishing e...
research
01/22/2019

Linearized Multi-Sampling for Differentiable Image Transformation

We propose a novel image sampling method for differentiable image transf...
research
12/06/2021

Interpretable Image Classification with Differentiable Prototypes Assignment

We introduce ProtoPool, an interpretable image classification model with...
research
03/21/2022

Training Quantised Neural Networks with STE Variants: the Additive Noise Annealing Algorithm

Training quantised neural networks (QNNs) is a non-differentiable optimi...
research
10/05/2022

Differentiable Mathematical Programming for Object-Centric Representation Learning

We propose topology-aware feature partitioning into k disjoint partition...
research
10/07/2019

Algorithmic Probability-guided Supervised Machine Learning on Non-differentiable Spaces

We show how complexity theory can be introduced in machine learning to h...

Please sign up or login with your details

Forgot password? Click here to reset