research
∙
09/15/2023
Scaling Laws for Sparsely-Connected Foundation Models
We explore the impact of parameter sparsity on the scaling behavior of T...
share
research
∙
08/03/2023
Accurate Neural Network Pruning Requires Rethinking Sparse Optimization
Obtaining versions of deep neural networks that are both highly-accurate...
research
∙
07/07/2023
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models
We present ongoing work on a new automatic code generation approach for ...
research
∙
06/05/2023
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
Recent advances in large language model (LLM) pretraining have led to hi...
research
∙
04/27/2023
JaxPruner: A concise library for sparsity research
This paper introduces JaxPruner, an open-source JAX-based pruning and sp...
research
∙
03/25/2023
Vision Models Can Be Efficiently Specialized via Few-Shot Task-Aware Compression
Recent vision architectures and self-supervised training methods enable ...
research
∙
02/07/2023
ZipLM: Hardware-Aware Structured Pruning of Language Models
The breakthrough performance of large language models (LLMs) comes with ...
research
∙
01/02/2023
Massive Language Models Can Be Accurately Pruned in One-Shot
We show for the first time that large-scale generative pretrained transf...
research
∙
10/31/2022
L-GreCo: An Efficient and General Framework for Layerwise-Adaptive Gradient Compression
Data-parallel distributed training of deep neural networks (DNN) has gai...
research
∙
10/31/2022
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Generative Pre-trained Transformer (GPT) models set themselves apart thr...
research
∙
10/14/2022
oViT: An Accurate Second-Order Pruning Framework for Vision Transformers
Models from the Vision Transformer (ViT) family have recently provided b...
research
∙
08/24/2022
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
We consider the problem of model compression for deep neural networks (D...
research
∙
03/14/2022
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
Pre-trained Transformer-based language models have become a key building...
research
∙
01/31/2022
SPDY: Accurate Pruning with Speedup Guarantees
The recent focus on the efficiency of deep neural networks (DNNs) has le...
research
∙
07/07/2021
Efficient Matrix-Free Approximations of Second-Order Information, with Applications to Pruning and Optimization
Efficiently approximating local curvature information of the loss functi...
research
∙
02/24/2020