Parameter Efficient Deep Neural Networks with Bilinear Projections

11/03/2020
โˆ™
by   Litao Yu, et al.
โˆ™
0
โˆ™

Recent research on deep neural networks (DNNs) has primarily focused on improving the model accuracy. Given a proper deep learning framework, it is generally possible to increase the depth or layer width to achieve a higher level of accuracy. However, the huge number of model parameters imposes more computational and memory usage overhead and leads to the parameter redundancy. In this paper, we address the parameter redundancy problem in DNNs by replacing conventional full projections with bilinear projections. For a fully-connected layer with D input nodes and D output nodes, applying bilinear projection can reduce the model space complexity from ๐’ช(D^2) to ๐’ช(2D), achieving a deep model with a sub-linear layer size. However, structured projection has a lower freedom of degree compared to the full projection, causing the under-fitting problem. So we simply scale up the mapping size by increasing the number of output channels, which can keep and even boosts the model accuracy. This makes it very parameter-efficient and handy to deploy such deep models on mobile systems with memory limitations. Experiments on four benchmark datasets show that applying the proposed bilinear projection to deep neural networks can achieve even higher accuracies than conventional full DNNs, while significantly reduces the model size.

READ FULL TEXT
research
โˆ™ 02/11/2015

An exploration of parameter redundancy in deep networks with circulant projections

We explore the redundancy of parameters in deep neural networks by repla...
research
โˆ™ 12/22/2018

Random Projection in Deep Neural Networks

This work investigates the ways in which deep learning methods can benef...
research
โˆ™ 01/13/2022

Examining and Mitigating the Impact of Crossbar Non-idealities for Accurate Implementation of Sparse Deep Neural Networks

Recently several structured pruning techniques have been introduced for ...
research
โˆ™ 12/04/2017

Adaptive Quantization for Deep Neural Network

In recent years Deep Neural Networks (DNNs) have been rapidly developed ...
research
โˆ™ 02/24/2016

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

Recent research on deep neural networks has focused primarily on improvi...
research
โˆ™ 01/29/2020

Constructing Deep Neural Networks with a Priori Knowledge of Wireless Tasks

Deep neural networks (DNNs) have been employed for designing wireless sy...
research
โˆ™ 07/18/2023

Optimistic Estimate Uncovers the Potential of Nonlinear Models

We propose an optimistic estimate to evaluate the best possible fitting ...

Please sign up or login with your details

Forgot password? Click here to reset