Deep Learning with Data Dependent Implicit Activation Function

02/01/2018
by   Bao Wang, et al.
1

Though deep neural networks (DNNs) achieve remarkable performances in many artificial intelligence tasks, the lack of training instances remains a notorious challenge. As the network goes deeper, the generalization accuracy decays rapidly in the situation of lacking massive amounts of training data. In this paper, we propose novel deep neural network structures that can be inherited from all existing DNNs with almost the same level of complexity, and develop simple training algorithms. We show our paradigm successfully resolves the lack of data issue. Tests on the CIFAR10 and CIFAR100 image recognition datasets show that the new paradigm leads to 20% to 30% relative error rate reduction compared to their base DNNs. The intuition of our algorithms for deep residual network stems from theories of the partial differential equation (PDE) control problems. Code will be made available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/18/2023

Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks

Deep neural networks (DNNs) are often trained on the premise that the co...
research
10/08/2020

Approximating smooth functions by deep neural networks with sigmoid activation function

We study the power of deep neural networks (DNNs) with sigmoid activatio...
research
12/10/2019

Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

Motivated by the gap between theoretical optimal approximation rates of ...
research
03/10/2021

Partial Differential Equations is All You Need for Generating Neural Architectures – A Theory for Physical Artificial Intelligence Systems

In this work, we generalize the reaction-diffusion equation in statistic...
research
07/07/2018

Anytime Neural Prediction via Slicing Networks Vertically

The pioneer deep neural networks (DNNs) have emerged to be deeper or wid...
research
07/16/2019

Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning

Improving the accuracy and robustness of deep neural nets (DNNs) and ada...
research
02/13/2018

Barista - a Graphical Tool for Designing and Training Deep Neural Networks

In recent years, the importance of deep learning has significantly incre...

Please sign up or login with your details

Forgot password? Click here to reset