Heuristic Rank Selection with Progressively Searching Tensor Ring Network

09/22/2020
by   Nannan Li, et al.
5

Recently, Tensor Ring Networks (TRNs) have been applied in deep networks, achieving remarkable successes in compression ratio and accuracy. Although highly related to the performance of TRNs, rank is seldom studied in previous works and usually set to equal in experiments. Meanwhile, there is not any heuristic method to choose the rank, and an enumerating way to find appropriate rank is extremely time-consuming. Interestingly, we discover that part of the rank elements is sensitive and usually aggregate in a certain region, namely an interest region. Therefore, based on the above phenomenon, we propose a novel progressive genetic algorithm named Progressively Searching Tensor Ring Network Search (PSTRN), which has the ability to find optimal rank precisely and efficiently. Through the evolutionary phase and progressive phase, PSTRN can converge to the interest region quickly and harvest good performance. Experimental results show that PSTRN can significantly reduce the complexity of seeking rank, compared with the enumerating method. Furthermore, our method is validated on public benchmarks like MNIST, CIFAR10/100 and HMDB51, achieving state-of-the-art performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/27/2022

Bayesian Robust Tensor Ring Model for Incomplete Multiway Data

Low-rank tensor completion aims to recover missing entries from the obse...
research
04/22/2020

Hierarchical Tensor Ring Completion

Tensor completion can estimate missing values of a high-order data from ...
research
09/30/2021

Semi-tensor Product-based TensorDecomposition for Neural Network Compression

The existing tensor networks adopt conventional matrix product for conne...
research
11/09/2018

The trouble with tensor ring decompositions

The tensor train decomposition decomposes a tensor into a "train" of 3-w...
research
09/06/2021

Fast Hypergraph Regularized Nonnegative Tensor Ring Factorization Based on Low-Rank Approximation

For the high dimensional data representation, nonnegative tensor ring (N...
research
05/20/2017

Learning Efficient Tensor Representations with Ring Structure Networks

Tensor train (TT) decomposition is a powerful representation for high-or...

Please sign up or login with your details

Forgot password? Click here to reset