DeepAI AI Chat
Log In Sign Up

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

by   Alexandra Chronopoulou, et al.
National Technical University of Athens

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.


Fine-tuned Language Models for Text Classification

Transfer learning has revolutionized computer vision, but existing appro...

Parameter-Efficient Transfer Learning with Diff Pruning

While task-specific finetuning of pretrained networks has led to signifi...

Reprogramming Language Models for Molecular Representation Learning

Recent advancements in transfer learning have made it a promising approa...

Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning

Recent research shows synthetic data as a source of supervision helps pr...

ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation

Joint speech-language training is challenging due to the large demand fo...

Gradient-Based Automated Iterative Recovery for Parameter-Efficient Tuning

Pretrained large language models (LLMs) are able to solve a wide variety...