The Trade-offs of Domain Adaptation for Neural Language Models

09/21/2021
by   Dan Iter, et al.
0

In this paper, we connect language model adaptation with concepts of machine learning theory. We consider a training setup with a large out-of-domain set and a small in-domain set. As a first contribution, we derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distribution. As a second contribution, we present how the most popular data selection techniques – importance sampling, intelligent data selection and influence functions – can be presented in a common framework which highlights their similarity and also their subtle differences.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/05/2020

Efficient MDI Adaptation for n-gram Language Models

This paper presents an efficient algorithm for n-gram language model ada...
research
12/20/2014

Incremental Adaptation Strategies for Neural Network Language Models

It is today acknowledged that neural network language models outperform ...
research
04/22/2020

Moment-Based Domain Adaptation: Learning Bounds and Algorithms

This thesis contributes to the mathematical foundation of domain adaptat...
research
03/14/2022

From Big to Small: Adaptive Learning to Partial-Set Domains

Domain adaptation targets at knowledge acquisition and dissemination fro...
research
09/14/2018

Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation

To better understand the effectiveness of continued training, we analyze...
research
06/12/2023

In-context Cross-Density Adaptation on Noisy Mammogram Abnormalities Detection

This paper investigates the impact of breast density distribution on the...
research
05/28/2021

Data Acquisition for Improving Machine Learning Models

The vast advances in Machine Learning over the last ten years have been ...

Please sign up or login with your details

Forgot password? Click here to reset