MDMLP: Image Classification from Scratch on Small Datasets with MLP

05/28/2022
by   Tian Lv, et al.
0

The attention mechanism has become a go-to technique for natural language processing and computer vision tasks. Recently, the MLP-Mixer and other MLP-based architectures, based simply on multi-layer perceptrons (MLPs), are also powerful compared to CNNs and attention techniques and raises a new research direction. However, the high capability of the MLP-based networks severely relies on large volume of training data, and lacks of explanation ability compared to the Vision Transformer (ViT) or ConvNets. When trained on small datasets, they usually achieved inferior results than ConvNets. To resolve it, we present (i) multi-dimensional MLP (MDMLP), a conceptually simple and lightweight MLP-based architecture yet achieves SOTA when training from scratch on small-size datasets; (ii) multi-dimension MLP Attention Tool (MDAttnTool), a novel and efficient attention mechanism based on MLPs. Even without strong data augmentation, MDMLP achieves 90.90 with only 0.3M parameters, while the well-known MLP-Mixer achieves 85.45 17.1M parameters. In addition, the lightweight MDAttnTool highlights objects in images, indicating its explanation power. Our code is available at https://github.com/Amoza-Theodore/MDMLP.

READ FULL TEXT
research
06/25/2021

Vision Transformer Architecture Search

Recently, transformers have shown great superiority in solving computer ...
research
09/20/2023

AttentionMix: Data augmentation method that relies on BERT attention mechanism

The Mixup method has proven to be a powerful data augmentation technique...
research
07/26/2023

Consensus-Adaptive RANSAC

RANSAC and its variants are widely used for robust estimation, however, ...
research
03/15/2021

UPANets: Learning from the Universal Pixel Attention Networks

Among image classification, skip and densely-connection-based networks h...
research
04/28/2022

Attention Mechanism with Energy-Friendly Operations

Attention mechanism has become the dominant module in natural language p...
research
08/03/2021

Vision Transformer with Progressive Sampling

Transformers with powerful global relation modeling abilities have been ...
research
01/26/2022

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

Attention mechanism has been widely believed as the key to success of vi...

Please sign up or login with your details

Forgot password? Click here to reset