KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier

10/06/2021
by   Linyang Li, et al.
0

Pre-trained models are widely used in fine-tuning downstream tasks with linear classifiers optimized by the cross-entropy loss, which might face robustness and stability problems. These problems can be improved by learning representations that focus on similarities in the same class and contradictions in different classes when making predictions. In this paper, we utilize the K-Nearest Neighbors Classifier in pre-trained model fine-tuning. For this KNN classifier, we introduce a supervised momentum contrastive learning framework to learn the clustered representations of the supervised downstream tasks. Extensive experiments on text classification tasks and robustness tests show that by incorporating KNNs with the traditional fine-tuning process, we can obtain significant improvements on the clean accuracy in both rich-source and few-shot settings and can improve the robustness against adversarial attacks. [all codes is available at https://github.com/LinyangLee/KNN-BERT]

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/12/2020

Bi-tuning of Pre-trained Representations

It is common within the deep learning community to first pre-train a dee...
research
03/28/2020

Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

Pretrained models from self-supervision are prevalently used in fine-tun...
research
05/30/2022

Prompt-aligned Gradient for Prompt Tuning

Thanks to the large pre-trained vision-language models (VLMs) like CLIP,...
research
03/25/2022

Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks

While fine-tuning pre-trained models for downstream classification is th...
research
10/21/2021

Fast Model Editing at Scale

While large pre-trained models have enabled impressive results on a vari...
research
05/16/2023

Prompt-Tuning Decision Transformer with Preference Ranking

Prompt-tuning has emerged as a promising method for adapting pre-trained...
research
08/11/2021

Learning Oculomotor Behaviors from Scanpath

Identifying oculomotor behaviors relevant for eye-tracking applications ...

Please sign up or login with your details

Forgot password? Click here to reset