DeepAI AI Chat
Log In Sign Up

Universal Approximation with Quadratic Deep Networks

by   Fenglei Fan, et al.
Rensselaer Polytechnic Institute

Recently, deep learning has been playing a central role in machine learning research and applications. Since AlexNet, increasingly more advanced networks have achieved state-of-the-art performance in computer vision, speech recognition, language processing, game playing, medical imaging, and so on. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant second-order neuron enjoys an enhanced expressive capability over the conventional neuron. However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been studied up to now, preferably in relation to that of a conventional neural network. In this paper, we ask three basic questions regarding the expressive capability of a quadratic network: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight into universal approximation? Our main contributions are the three theorems shedding light upon these three questions and demonstrating the merits of a quadratic network in terms of expressive efficiency, unique capability, and compact architecture respectively.


One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks

Inspired by neuronal diversity in the biological neural system, a pletho...

Fuzzy Logic Interpretation of Artificial Neural Networks

Over past several years, deep learning has achieved huge successes in va...

Quadratic Autoencoder for Low-Dose CT Denoising

Recently, deep learning has transformed many fields including medical im...

Expressivity and Trainability of Quadratic Networks

Inspired by diversity of biological neurons, quadratic artificial neuron...

General Backpropagation Algorithm for Training Second-order Neural Networks

The artificial neural network is a popular framework in machine learning...

Exploring Transfer Function Nonlinearity in Echo State Networks

Supralinear and sublinear pre-synaptic and dendritic integration is cons...

QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Exploration

The significant success of Deep Neural Networks (DNNs) is highly promote...