Lifting Layers: Analysis and Applications

03/23/2018
by   Peter Ochs, et al.
0

The great advances of learning-based approaches in image processing and computer vision are largely based on deeply nested networks that compose linear transfer functions with suitable non-linearities. Interestingly, the most frequently used non-linearities in imaging applications (variants of the rectified linear unit) are uncommon in low dimensional approximation problems. In this paper we propose a novel non-linear transfer function, called lifting, which is motivated from a related technique in convex optimization. A lifting layer increases the dimensionality of the input, naturally yields a linear spline when combined with a fully connected layer, and therefore closes the gap between low and high dimensional approximation problems. Moreover, applying the lifting operation to the loss layer of the network allows us to handle non-convex and flat (zero-gradient) cost functions. We analyze the proposed lifting theoretically, exemplify interesting properties in synthetic experiments and demonstrate its effectiveness in deep learning approaches to image classification and denoising.

READ FULL TEXT

page 2

page 12

12/21/2017

Non-convex Optimization for Machine Learning

A vast majority of machine learning algorithms train their models and pe...
12/04/2017

An Equivalence of Fully Connected Layer and Convolutional Layer

This article demonstrates that convolutional operation can be converted ...
05/17/2019

SSFN: Self Size-estimating Feed-forward Network and Low Complexity Design

We design a self size-estimating feed-forward network (SSFN) using a joi...
06/09/2022

Redundancy in Deep Linear Neural Networks

Conventional wisdom states that deep linear neural networks benefit from...
07/21/2022

Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning

Optimizing noisy functions online, when evaluating the objective require...
10/16/2020

Strengthened SDP Verification of Neural Network Robustness via Non-Convex Cuts

There have been major advances on the design of neural networks, but sti...
06/04/2019

An Introduction to Deep Morphological Networks

The recent impressive results of deep learning-based methods on computer...