Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement

03/15/2023
by   Fartash Faghri, et al.
0

We propose Dataset Reinforcement, a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no additional training cost for users. We propose a Dataset Reinforcement strategy based on data augmentation and knowledge distillation. Our generic strategy is designed based on extensive analysis across CNN- and transformer-based models and performing large-scale study of distillation with state-of-the-art models with various data augmentations. We create a reinforced version of the ImageNet training dataset, called ImageNet+, as well as reinforced datasets CIFAR-100+, Flowers-102+, and Food-101+. Models trained with ImageNet+ are more accurate, robust, and calibrated, and transfer well to downstream tasks (e.g., segmentation and detection). As an example, the accuracy of ResNet-50 improves by 1.7 ImageNetV2, and 10.0 ImageNet validation set is also reduced by 9.9 Mask-RCNN for object detection on MS-COCO, the mean average precision improves by 0.8 For MobileNetV3 and Swin-Tiny we observe significant improvements on ImageNet-R/A/C of up to 10 and fine-tuned on CIFAR-100+, Flowers-102+, and Food-101+, reach up to 3.4 improved accuracy.

READ FULL TEXT

page 5

page 12

page 20

page 22

research
04/08/2022

Does Robustness on ImageNet Transfer to Downstream Tasks?

As clean ImageNet accuracy nears its ceiling, the research community is ...
research
02/13/2020

CEB Improves Model Robustness

We demonstrate that the Conditional Entropy Bottleneck (CEB) can improve...
research
09/18/2023

Heterogeneous Generative Knowledge Distillation with Masked Image Modeling

Small CNN-based models usually require transferring knowledge from a lar...
research
11/15/2017

Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

Deep learning networks have achieved state-of-the-art accuracies on comp...
research
09/30/2019

RandAugment: Practical data augmentation with no separate search

Recent work has shown that data augmentation has the potential to signif...
research
05/25/2023

VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale

The tremendous success of large models trained on extensive datasets dem...
research
04/12/2023

NutritionVerse-Thin: An Optimized Strategy for Enabling Improved Rendering of 3D Thin Food Models

With the growth in capabilities of generative models, there has been gro...

Please sign up or login with your details

Forgot password? Click here to reset