T-Basis: a Compact Representation for Neural Networks

07/13/2020
by   Anton Obukhov, et al.
20

We introduce T-Basis, a novel concept for a compact representation of a set of tensors, each of an arbitrary shape, which is often seen in Neural Networks. Each of the tensors in the set is modeled using Tensor Rings, though the concept applies to other Tensor Networks. Owing its name to the T-shape of nodes in diagram notation of Tensor Rings, T-Basis is simply a list of equally shaped three-dimensional tensors, used to represent Tensor Ring nodes. Such representation allows us to parameterize the tensor set with a small number of parameters (coefficients of the T-Basis tensors), scaling logarithmically with each tensor's size in the set and linearly with the dimensionality of T-Basis. We evaluate the proposed approach on the task of neural network compression and demonstrate that it reaches high compression rates at acceptable performance drops. Finally, we analyze memory and operation requirements of the compressed networks and conclude that T-Basis networks are equally well suited for training and inference in resource-constrained environments and usage on the edge devices.

READ FULL TEXT

page 8

page 9

02/14/2020

Back-and-Forth prediction for deep tensor compression

Recent AI applications such as Collaborative Intelligence with neural ne...
03/05/2022

How to Train Unstable Looped Tensor Network

A rising problem in the compression of Deep Neural Networks is how to re...
01/15/2022

Edge-based Tensor prediction via graph neural networks

Message-passing neural networks (MPNN) have shown extremely high efficie...
11/12/2021

Nonlinear Tensor Ring Network

The state-of-the-art deep neural networks (DNNs) have been widely applie...
07/11/2019

Randomized Functional Sparse Tucker Tensor for Compression and Fast Visualization of Scientific Data

We propose a strategy to compress and store large volumes of scientific ...
10/10/2020

Block-term Tensor Neural Networks

Deep neural networks (DNNs) have achieved outstanding performance in a w...
10/28/2018

Smoothed Analysis of Discrete Tensor Decomposition and Assemblies of Neurons

We analyze linear independence of rank one tensors produced by tensor po...

Code Repositories

tbasis

T-Basis: a Compact Representation for Neural Networks


view repo