PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models

06/16/2020
by   Eyal Ben-David, et al.
0

Pivot-based neural representation models have lead to significant progress in domain adaptation for NLP. However, previous works that follow this approach utilize only labeled data from the source domain and unlabeled data from the source and target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. To alleviate this, we propose PERL: A representation learning model that extends contextualized word embedding models such as BERT with pivot-based fine-tuning. PERL outperforms strong baselines across 22 sentiment classification domain adaptation setups, improves in-domain model performance, yields effective reduced-size models and increases model stability.

READ FULL TEXT
research
09/01/2021

DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extraction

The rise of pre-trained language models has yielded substantial progress...
research
04/30/2020

Vocabulary Adaptation for Distant Domain Adaptation in Neural Machine Translation

Neural machine translation (NMT) models do not work well in domains diff...
research
07/30/2023

Open-Set Domain Adaptation with Visual-Language Foundation Models

Unsupervised domain adaptation (UDA) has proven to be very effective in ...
research
10/05/2016

Neural Structural Correspondence Learning for Domain Adaptation

Domain adaptation, adapting models from domains rich in labeled training...
research
10/06/2022

Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation

Prompt tuning, or the conditioning of a frozen pretrained language model...
research
12/20/2022

StyleDomain: Analysis of StyleSpace for Domain Adaptation of StyleGAN

Domain adaptation of GANs is a problem of fine-tuning the state-of-the-a...
research
04/24/2023

Multi-Source to Multi-Target Decentralized Federated Domain Adaptation

Heterogeneity across devices in federated learning (FL) typically refers...

Please sign up or login with your details

Forgot password? Click here to reset