Spectral Tensor Train Parameterization of Deep Learning Layers

03/07/2021
by   Anton Obukhov, et al.
15

We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting and both compression and improved training stability in the generative adversarial training setting.

READ FULL TEXT

page 4

page 6

page 8

10/04/2018

Learning Compressed Transforms with Low Displacement Rank

The low displacement rank (LDR) framework for structured matrices repres...
06/03/2013

Predicting Parameters in Deep Learning

We demonstrate that there is significant redundancy in the parameterizat...
05/22/2018

Low-Rank Tensor Decomposition via Multiple Reshaping and Reordering Operations

Tensor decomposition has been widely applied to find low-rank representa...
07/30/2022

Distilled Low Rank Neural Radiance Field with Quantization for Light Field Compression

In this paper, we propose a novel light field compression method based o...
08/13/2021

FedPara: Low-rank Hadamard Product Parameterization for Efficient Federated Learning

To overcome the burdens on frequent model uploads and downloads during f...
09/24/2017

Tensor-Based Classifiers for Hyperspectral Data Analysis

In this work, we present tensor-based linear and nonlinear models for hy...
08/04/2017

A Low-rank Spline Approximation of Planar Domains

Construction of spline surfaces from given boundary curves is one of the...

Code Repositories

torch-householder

Efficient Householder Transformation in PyTorch


view repo

sttp

Spectral Tensor Train Parameterization of Deep Learning Layers


view repo