On the Importance of Data Size in Probing Fine-tuned Models

03/17/2022
by   Houman Mehrafarin, et al.
0

Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model's linguistic knowledge.

READ FULL TEXT

page 5

page 6

page 8

page 11

research
10/06/2020

On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers

Fine-tuning pre-trained contextualized embedding models has become an in...
research
09/10/2021

How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy

It is widely accepted that fine-tuning pre-trained language models usual...
research
09/15/2021

Can Edge Probing Tasks Reveal Linguistic Knowledge in QA Models?

There have been many efforts to try to understand what gram-matical know...
research
10/14/2022

Self-Repetition in Abstractive Neural Summarizers

We provide a quantitative and qualitative analysis of self-repetition in...
research
04/02/2023

Instance-level Trojan Attacks on Visual Question Answering via Adversarial Learning in Neuron Activation Space

Malicious perturbations embedded in input data, known as Trojan attacks,...
research
10/21/2022

On-Device Model Fine-Tuning with Label Correction in Recommender Systems

To meet the practical requirements of low latency, low cost, and good pr...
research
09/05/2017

Fine-tuning deep CNN models on specific MS COCO categories

Fine-tuning of a deep convolutional neural network (CNN) is often desire...

Please sign up or login with your details

Forgot password? Click here to reset