EAPruning: Evolutionary Pruning for Vision Transformers and CNNs

10/01/2022
by   Qingyuan Li, et al.
0

Structured pruning greatly eases the deployment of large neural networks in resource-constrained environments. However, current methods either involve strong domain expertise, require extra hyperparameter tuning, or are restricted only to a specific type of network, which prevents pervasive industrial applications. In this paper, we undertake a simple and effective approach that can be easily applied to both vision transformers and convolutional neural networks. Specifically, we consider pruning as an evolution process of sub-network structures that inherit weights through reconstruction techniques. We achieve a 50 and 1.34x speedup respectively. For DeiT-Base, we reach nearly 40 reduction and 1.4x speedup. Our code will be made available.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/20/2017

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration

Although deep Convolutional Neural Network (CNN) has shown better perfor...
research
06/13/2022

Leveraging Structured Pruning of Convolutional Neural Networks

Structured pruning is a popular method to reduce the cost of convolution...
research
01/13/2023

GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer

The recently proposed Vision transformers (ViTs) have shown very impress...
research
02/04/2023

Knowledge Distillation in Vision Transformers: A Critical Review

In Natural Language Processing (NLP), Transformers have already revoluti...
research
05/20/2021

A Probabilistic Approach to Neural Network Pruning

Neural network pruning techniques reduce the number of parameters withou...
research
06/17/2019

Structured Pruning of Recurrent Neural Networks through Neuron Selection

Recurrent neural networks (RNNs) have recently achieved remarkable succe...
research
03/30/2023

SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer

High-resolution images enable neural networks to learn richer visual rep...

Please sign up or login with your details

Forgot password? Click here to reset