Low-Rank Tensor Decomposition via Multiple Reshaping and Reordering Operations

05/22/2018 ∙ by Chao Li, et al. ∙ 0

Tensor decomposition has been widely applied to find low-rank representations for real-world data and more recently for neural-network parameters too. For the latter, the unfolded matrices may not always be low-rank because the modes of the parameter tensor do not usually have any physical meaning that can be exploited for efficiency. This raises the following question: how can we find low-rank structures when the tensor modes do not have any physical meaning associated with them? For this purpose, we propose a new decomposition method in this paper. Our method uses reshaping and reordering operations that are strictly more general than the unfolding operation. These operations enable us to discover new low-rank structures that are beyond the reach of existing tensor methods. We prove an important theoretical result establishing conditions under which our method results in a unique solution. The experimental results confirm the correctness of our theoretical works and the effectiveness of our methods for weight compression in deep neural networks.



page 9

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.