Log In Sign Up

A Broad Study of Pre-training for Domain Generalization and Adaptation

by   Donghyun Kim, et al.

Deep models must learn robust and transferable representations in order to perform well on new domains. While domain transfer methods (e.g., domain adaptation, domain generalization) have been proposed to learn transferable representations across domains, they are typically applied to ResNet backbones pre-trained on ImageNet. Thus, existing works pay little attention to the effects of pre-training on domain transfer tasks. In this paper, we provide a broad study and in-depth analysis of pre-training for domain adaptation and generalization, namely: network architectures, size, pre-training loss, and datasets. We observe that simply using a state-of-the-art backbone outperforms existing state-of-the-art domain adaptation baselines and set new baselines on Office-Home and DomainNet improving by 10.7% and 5.5%. We hope that this work can provide more insights for future domain transfer research.


page 1

page 2

page 3

page 4


AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization

State-of-the-art abstractive summarization models generally rely on exte...

Pre-train or Annotate? Domain Adaptation with a Constrained Budget

Recent work has demonstrated that pre-training in-domain language models...

CrossNER: Evaluating Cross-Domain Named Entity Recognition

Cross-domain named entity recognition (NER) models are able to cope with...

Modified Distribution Alignment for Domain Adaptation with Pre-trainedInception ResNet

Deep neural networks have been widely used in computer vision. There are...

Convergence of gradient based pre-training in Denoising autoencoders

The success of deep architectures is at least in part attributed to the ...

A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization

We study the problem of domain adaptation for neural abstractive summari...

Temporal Effects on Pre-trained Models for Language Processing Tasks

Keeping the performance of language technologies optimal as time passes ...