Extrapolating from a Single Image to a Thousand Classes using Distillation

12/01/2021
by   Yuki M. Asano, et al.
15

What can neural networks learn about the visual world from a single image? While it obviously cannot contain the multitudes of possible objects, scenes and lighting conditions that exist - within the space of all possible 256^(3x224x224) 224-sized square images, it might still provide a strong prior for natural images. To analyze this hypothesis, we develop a framework for training neural networks from scratch using a single image by means of knowledge distillation from a supervisedly pretrained teacher. With this, we find that the answer to the above question is: 'surprisingly, a lot'. In quantitative terms, we find top-1 accuracies of 94 ImageNet and, by extending this method to audio, 84 extensive analyses we disentangle the effect of augmentations, choice of source image and network architectures and also discover "panda neurons" in networks that have never seen a panda. This work shows that one image can be used to extrapolate to thousands of object classes and motivates a renewed research agenda on the fundamental interplay of augmentations and image.

READ FULL TEXT

page 1

page 8

page 12

page 14

page 15

page 16

research
11/22/2021

Local-Selective Feature Distillation for Single Image Super-Resolution

Recent improvements in convolutional neural network (CNN)-based single i...
research
04/26/2020

DGD: Densifying the Knowledge of Neural Networks with Filter Grafting and Knowledge Distillation

With a fixed model structure, knowledge distillation and filter grafting...
research
04/29/2022

Multiple Degradation and Reconstruction Network for Single Image Denoising via Knowledge Distillation

Single image denoising (SID) has achieved significant breakthroughs with...
research
06/12/2018

Knowledge Distillation by On-the-Fly Native Ensemble

Knowledge distillation is effective to train small and generalisable net...
research
09/11/2020

Extending Label Smoothing Regularization with Self-Knowledge Distillation

Inspired by the strong correlation between the Label Smoothing Regulariz...
research
08/22/2019

Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement

Generic Image recognition is a fundamental and fairly important visual p...
research
04/30/2020

Sequence Information Channel Concatenation for Improving Camera Trap Image Burst Classification

Camera Traps are extensively used to observe wildlife in their natural h...

Please sign up or login with your details

Forgot password? Click here to reset