DeepAI AI Chat
Log In Sign Up

Effect of large-scale pre-training on full and few-shot transfer learning for natural and medical images

by   Mehdi Cherti, et al.

Transfer learning aims to exploit pre-trained models for more efficient follow-up training on wide range of downstream tasks and datasets, enabling successful training also on small data. Recent line of work posits strong benefits for model generalization and transfer when model size, data size, and compute budget are increased for the pre-training. It remains however still largely unclear whether the observed transfer improvement due to increase in scale also holds when source and target data distributions are far apart from each other. In this work we conduct large-scale pre-training on large source datasets of either natural (ImageNet-21k/1k) or medical chest X-Ray images and compare full and few-shot transfer using different target datasets from both natural and medical imaging domains. Our observations provide evidence that while pre-training and transfer on closely related datasets do show clear benefit of increasing model and data size during pre-training, such benefits are not clearly visible when source and target datasets are further apart. These observations hold across both full and few-shot transfer and indicate that scaling laws hinting improvement of generalization and transfer with increasing model and data size are incomplete and should also take into account the degree of how distinct the source and target data distributions are, to correctly predict effect of model size and data size variation during pre-training on transfer. (Repository for reproducing the experiments will be made available.)


page 7

page 15

page 16


Supervised Transfer Learning at Scale for Medical Imaging

Transfer learning is a standard technique to improve performance on task...

Self-training for Few-shot Transfer Across Extreme Task Differences

All few-shot learning techniques must be pre-trained on a large, labeled...

Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers

Empirical science of neural scaling laws is a rapidly growing area of si...

Understanding Cross-Domain Few-Shot Learning: An Experimental Study

Cross-domain few-shot learning has drawn increasing attention for handli...

Beyond Flatland: Pre-training with a Strong 3D Inductive Bias

Pre-training on large-scale databases consisting of natural images and t...

TransferD2: Automated Defect Detection Approach in Smart Manufacturing using Transfer Learning Techniques

Quality assurance is crucial in the smart manufacturing industry as it i...

Resource-efficient domain adaptive pre-training for medical images

The deep learning-based analysis of medical images suffers from data sca...