DeepAI AI Chat
Log In Sign Up

AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning

by   Yuhan Liu, et al.

With the rapid adoption of machine learning (ML), a number of domains now use the approach of fine-tuning models pre-trained on a large corpus of data. However, our experiments show that even fine-tuning on models like BERT can take many hours when using GPUs. While prior work proposes limiting the number of layers that are fine-tuned, e.g., freezing all layers but the last layer, we find that such static approaches lead to reduced accuracy. We propose, AutoFreeze, a system that uses an adaptive approach to choose which layers are trained and show how this can accelerate model fine-tuning while preserving accuracy. We also develop mechanisms to enable efficient caching of intermediate activations which can reduce the forward computation time when performing fine-tuning. Our evaluation on fourNLP tasks shows that AutoFreeze, with caching enabled, can improve fine-tuning performance by up to 2.55x.


page 1

page 2

page 3

page 4


How fine can fine-tuning be? Learning efficient language models

State-of-the-art performance on language understanding tasks is now achi...

Surgical Fine-Tuning Improves Adaptation to Distribution Shifts

A common approach to transfer learning under distribution shift is to fi...

lo-fi: distributed fine-tuning without communication

When fine-tuning large neural networks, it is common to use multiple nod...

Improving Learning-to-Defer Algorithms Through Fine-Tuning

The ubiquity of AI leads to situations where humans and AI work together...

CIFT: Crowd-Informed Fine-Tuning to Improve Machine Learning Ability

Item Response Theory (IRT) allows for measuring ability of Machine Learn...

The Weighted Tsetlin Machine: Compressed Representations with Weighted Clauses

The Tsetlin Machine (TM) is an interpretable mechanism for pattern recog...