Unsupervised Finetuning

10/18/2021
by   Suichan Li, et al.
14

This paper studies "unsupervised finetuning", the symmetrical problem of the well-known "supervised finetuning". Given a pretrained model and small-scale unlabeled target data, unsupervised finetuning is to adapt the representation pretrained from the source domain to the target domain so that better transfer performance can be obtained. This problem is more challenging than the supervised counterpart, as the low data density in the small-scale target data is not friendly for unsupervised learning, leading to the damage of the pretrained representation and poor representation in the target domain. In this paper, we find the source data is crucial when shifting the finetuning paradigm from supervise to unsupervise, and propose two simple and effective strategies to combine source and target data into unsupervised finetuning: "sparse source data replaying", and "data mixing". The motivation of the former strategy is to add a small portion of source data back to occupy their pretrained representation space and help push the target data to reside in a smaller compact space; and the motivation of the latter strategy is to increase the data density and help learn more compact representation. To demonstrate the effectiveness of our proposed “unsupervised finetuning” strategy, we conduct extensive experiments on multiple different target datasets, which show better transfer performance than the naive strategy.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/01/2020

Sequential Unsupervised Domain Adaptation through Prototypical Distributions

We develop an algorithm for unsupervised domain adaptation (UDA) of a cl...
research
10/29/2019

Weakly-Supervised Deep Learning for Domain Invariant Sentiment Classification

The task of learning a sentiment classification model that adapts well t...
research
06/03/2015

Unsupervised domain adaption dictionary learning for visual recognition

Over the last years, dictionary learning method has been extensively app...
research
02/08/2023

A Prototype-Oriented Clustering for Domain Shift with Source Privacy

Unsupervised clustering under domain shift (UCDS) studies how to transfe...
research
04/11/2019

Deep Transfer Learning for Single-Channel Automatic Sleep Staging with Channel Mismatch

Many sleep studies suffer from the problem of insufficient data to fully...
research
12/04/2020

Few-shot Image Generation with Elastic Weight Consolidation

Few-shot image generation seeks to generate more data of a given domain,...
research
07/25/2017

Representation Learning on Large and Small Data

Deep learning owes its success to three key factors: scale of data, enha...

Please sign up or login with your details

Forgot password? Click here to reset