Exploiting Local Structures with the Kronecker Layer in Convolutional Networks

12/31/2015
by   Shuchang Zhou, et al.
0

In this paper, we propose and study a technique to reduce the number of parameters and computation time in convolutional neural networks. We use Kronecker product to exploit the local structures within convolution and fully-connected layers, by replacing the large weight matrices by combinations of multiple Kronecker products of smaller matrices. Just as the Kronecker product is a generalization of the outer product from vectors to matrices, our method is a generalization of the low rank approximation method for convolution neural networks. We also introduce combinations of different shapes of Kronecker product to increase modeling capacity. Experiments on SVHN, scene text recognition and ImageNet dataset demonstrate that we can achieve 3.3 × speedup or 3.6 × parameter reduction with less than 1% drop in accuracy, showing the effectiveness and efficiency of our method. Moreover, the computation efficiency of Kronecker layer makes using larger feature map possible, which in turn enables us to outperform the previous state-of-the-art on both SVHN(digit recognition) and CASIA-HWDB (handwritten Chinese character recognition) datasets.

READ FULL TEXT
research
07/21/2015

Compression of Fully-Connected Layer in Neural Network by Kronecker Product

In this paper we propose and study a technique to reduce the number of p...
research
04/30/2019

PR Product: A Substitute for Inner Product in Neural Networks

In this paper, we analyze the inner product of weight vector and input v...
research
07/30/2015

Multilinear Map Layer: Prediction Regularization by Structural Constraint

In this paper we propose and study a technique to impose structural cons...
research
03/24/2023

Factorizers for Distributed Sparse Block Codes

Distributed sparse block codes (SBCs) exhibit compact representations fo...
research
04/04/2018

Building Efficient CNN Architecture for Offline Handwritten Chinese Character Recognition

Deep convolutional networks based methods have brought great breakthroug...
research
10/04/2018

Learning Compressed Transforms with Low Displacement Rank

The low displacement rank (LDR) framework for structured matrices repres...
research
11/05/2019

Dynamic Time Warp Convolutional Networks

Where dealing with temporal sequences it is fair to assume that the same...

Please sign up or login with your details

Forgot password? Click here to reset