SeKron: A Decomposition Method Supporting Many Factorization Structures

While convolutional neural networks (CNNs) have become the de facto standard for most image processing and computer vision applications, their deployment on edge devices remains challenging. Tensor decomposition methods provide a means of compressing CNNs to meet the wide range of device constraints by imposing certain factorization structures on their convolution tensors. However, being limited to the small set of factorization structures presented by state-of-the-art decomposition approaches can lead to sub-optimal performance. We propose SeKron, a novel tensor decomposition method that offers a wide variety of factorization structures, using sequences of Kronecker products. By recursively finding approximating Kronecker factors, we arrive at optimal decompositions for each of the factorization structures. We show that SeKron is a flexible decomposition that generalizes widely used methods, such as Tensor-Train (TT), Tensor-Ring (TR), Canonical Polyadic (CP) and Tucker decompositions. Crucially, we derive an efficient convolution projection algorithm shared by all SeKron structures, leading to seamless compression of CNN models. We validate SeKron for model compression on both high-level and low-level computer vision tasks and find that it outperforms state-of-the-art decomposition methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/13/2019

Einconv: Exploring Unexplored Tensor Decompositions for Convolutional Neural Networks

Tensor decomposition methods are one of the primary approaches for model...
research
04/26/2023

Tensor Decomposition for Model Reduction in Neural Networks: A Review

Modern neural networks have revolutionized the fields of computer vision...
research
07/03/2018

OCTen: Online Compression-based Tensor Decomposition

Tensor decompositions are powerful tools for large data analytics as the...
research
07/10/2018

Learning a Single Tucker Decomposition Network for Lossy Image Compression with Multiple Bits-Per-Pixel Rates

Lossy image compression (LIC), which aims to utilize inexact approximati...
research
09/29/2021

Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition

Modern Convolutional Neural Network (CNN) architectures, despite their s...
research
04/18/2023

Large-scale Dynamic Network Representation via Tensor Ring Decomposition

Large-scale Dynamic Networks (LDNs) are becoming increasingly important ...
research
10/08/2020

SWIFT: Scalable Wasserstein Factorization for Sparse Nonnegative Tensors

Existing tensor factorization methods assume that the input tensor follo...

Please sign up or login with your details

Forgot password? Click here to reset