Large Scale Learning of General Visual Representations for Transfer

12/24/2019
by   Alexander Kolesnikov, et al.
Google
10

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the weights on the target task. We scale up pre-training, and create a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes - from 10 to 1M labeled examples. BiT achieves 87.8 the Visual Task Adaptation Benchmark (which includes 19 tasks). On small datasets, BiT attains 86.4 97.6 the main components that lead to high transfer performance.

READ FULL TEXT

page 16

page 22

page 23

01/26/2021

Muppet: Massive Multi-task Representations with Pre-Finetuning

We propose pre-finetuning, an additional large-scale learning stage betw...
11/20/2020

Efficient Conditional Pre-training for Transfer Learning

Almost all the state-of-the-art neural networks for computer vision task...
05/05/2020

Multi-task pre-training of deep neural networks

In this work, we investigate multi-task learning as a way of pre-trainin...
09/28/2020

Scalable Transfer Learning with Expert Models

Transfer of pre-trained representations can improve sample efficiency an...
10/06/2021

Improving Fractal Pre-training

The deep neural networks used in modern computer vision systems require ...
05/02/2023

Transfer Visual Prompt Generator across LLMs

While developing a new vision-language LLM (VL-LLM) by pre-training on t...
05/05/2020

Multi-task pre-training of deep neural networks for digital pathology

In this work, we investigate multi-task learning as a way of pre-trainin...

Code Repositories

supervised_pretraining_GN_WS

Supervised NN for pre-training. with group normalization and weight standardization


view repo

A-Barebones-Image-Retrieval-System

This project presents a simple framework to retrieve images similar to a query image.


view repo

InDomainGeneralizationBenchmark

None


view repo

Big-Transfer-Learning

Used the Big Transfer Learning methods described by Google Research team in conjunction with PyTorch to replicate the behavior of an upstream dataset using the ResNet50-x3 architecture.


view repo

Please sign up or login with your details

Forgot password? Click here to reset