DeepAI AI Chat
Log In Sign Up

Stochastically Rank-Regularized Tensor Regression Networks

by   Arinbjörn Kolbeinsson, et al.

Over-parametrization of deep neural networks has recently been shown to be key to their successful training. However, it also renders them prone to overfitting and makes them expensive to store and train. Tensor regression networks significantly reduce the number of effective parameters in deep neural networks while retaining accuracy and the ease of training. They replace the flattening and fully-connected layers with a tensor regression layer, where the regression weights are expressed through the factors of a low-rank tensor decomposition. In this paper, to further improve tensor regression networks, we propose a novel stochastic rank-regularization. It consists of a novel randomized tensor sketching method to approximate the weights of tensor regression layers. We theoretically and empirically establish the link between our proposed stochastic rank-regularization and the dropout on low-rank tensor regression. Extensive experimental results with both synthetic data and real world datasets (i.e., CIFAR-100 and the UK Biobank brain MRI dataset) support that the proposed approach i) improves performance in both classification and regression tasks, ii) decreases overfitting, iii) leads to more stable training and iv) improves robustness to adversarial attacks and random noise.


page 1

page 2

page 3

page 4


Adaptive Low-Rank Regularization with Damping Sequences to Restrict Lazy Weights in Deep Networks

Overfitting is one of the critical problems in deep neural networks. Man...

Tensor Regression Networks with various Low-Rank Tensor Approximations

Tensor regression networks achieve high rate of compression of model par...

Adaptive Low-Rank Factorization to regularize shallow and deep neural networks

The overfitting is one of the cursing subjects in the deep learning fiel...

Comparison of echo state network output layer classification methods on noisy data

Echo state networks are a recently developed type of recurrent neural ne...

Defensive Tensorization

We propose defensive tensorization, an adversarial defence technique tha...

High-Dimensional Uncertainty Quantification via Rank- and Sample-Adaptive Tensor Regression

Fabrication process variations can significantly influence the performan...

Bayesian Tensorized Neural Networks with Automatic Rank Selection

Tensor decomposition is an effective approach to compress over-parameter...