Griffin: Rethinking Sparse Optimization for Deep Learning Architectures

07/27/2021
by   Jong Hoon Shin, et al.
0

This paper examines the design space trade-offs of DNNs accelerators aiming to achieve competitive performance and efficiency metrics for all four combinations of dense or sparse activation/weight tensors. To do so, we systematically examine the overheads of supporting sparsity on top of an optimized dense core. These overheads are modeled based on parameters that indicate how a multiplier can borrow a nonzero operation from the neighboring multipliers or future cycles. As a result of this exploration, we identify a few promising designs that perform better than prior work. Our findings suggest that even the best design targeting dual sparsity yields a 20 power efficiency when performing on single sparse models, i.e., those with only sparse weight or sparse activation tensors. We found that one can reuse resources of the same core to maintain high performance and efficiency when running single sparsity or dense models. We call this hybrid architecture Griffin. Griffin is 1.2, 3.0, 3.1, and 1.4X more power-efficient than state-of-the-art sparse architectures, for dense, weight-only sparse, activation-only sparse, and dual sparse models, respectively.

READ FULL TEXT

page 1

page 8

page 9

page 10

research
12/27/2021

Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks

In principle, sparse neural networks should be significantly more effici...
research
01/18/2022

Design Space Exploration of Dense and Sparse Mapping Schemes for RRAM Architectures

The impact of device and circuit-level effects in mixed-signal Resistive...
research
01/26/2023

SparDA: Accelerating Dynamic Sparse Deep Neural Networks via Sparse-Dense Transformation

Due to its high cost-effectiveness, sparsity has become the most importa...
research
05/20/2021

Dual-side Sparse Tensor Core

Leveraging sparsity in deep neural network (DNN) models is promising for...
research
07/15/2023

PASS: Exploiting Post-Activation Sparsity in Streaming Architectures for CNN Acceleration

With the ever-growing popularity of Artificial Intelligence, there is an...
research
02/02/2021

Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization

Training sparse networks to converge to the same performance as dense ne...
research
04/15/2023

STen: Productive and Efficient Sparsity in PyTorch

As deep learning models grow, sparsity is becoming an increasingly criti...

Please sign up or login with your details

Forgot password? Click here to reset