Vision Transformer Pruning Via Matrix Decomposition

by   Tianyi Sun, et al.

This is a further development of Vision Transformer Pruning via matrix decomposition. The purpose of the Vision Transformer Pruning is to prune the dimension of the linear projection of the dataset by learning their associated importance score in order to reduce the storage, run-time memory, and computational demands. In this paper we further reduce dimension and complexity of the linear projection by implementing and comparing several matrix decomposition methods while preserving the generated important features. We end up selected the Singular Value Decomposition as the method to achieve our goal by comparing the original accuracy scores in the original Github repository and the accuracy scores of using those matrix decomposition methods, including Singular Value Decomposition, four versions of QR Decomposition, and LU factorization.


Perturbation expansions and error bounds for the truncated singular value decomposition

Truncated singular value decomposition is a reduced version of the singu...

RSVD-CUR Decomposition for Matrix Triplets

We propose a restricted SVD based CUR (RSVD-CUR) decomposition for matri...

High Order Singular Value Decomposition for Plant Biodiversity Estimation

We propose a new method to estimate plant biodiversity with Rényi and Ra...

Visual Transformer Pruning

Visual transformer has achieved competitive performance on a variety of ...

Scalable Object Detection on Embedded Devices Using Weight Pruning and Singular Value Decomposition

This paper presents a method for optimizing object detection models by c...

Group Orbit Optimization: A Unified Approach to Data Normalization

In this paper we propose and study an optimization problem over a matrix...

Deviance Matrix Factorization

We investigate a general matrix factorization for deviance-based losses,...

Please sign up or login with your details

Forgot password? Click here to reset