Beyond One-hot Encoding: lower dimensional target embedding

06/28/2018
by   Pau Rodríguez, et al.
0

Target encoding plays a central role when learning Convolutional Neural Networks. In this realm, One-hot encoding is the most prevalent strategy due to its simplicity. However, this so widespread encoding schema assumes a flat label space, thus ignoring rich relationships existing among labels that can be exploited during training. In large-scale datasets, data does not span the full label space, but instead lies in a low-dimensional output manifold. Following this observation, we embed the targets into a low-dimensional space, drastically improving convergence speed while preserving accuracy. Our contribution is two fold: (i) We show that random projections of the label space are a valid tool to find such lower dimensional embeddings, boosting dramatically convergence rates at zero computational cost; and (ii) we propose a normalized eigenrepresentation of the class manifold that encodes the targets with minimal information loss, improving the accuracy of random projections encoding while enjoying the same convergence rates. Experiments on CIFAR-100, CUB200-2011, Imagenet, and MIT Places demonstrate that the proposed approach drastically improves convergence speed while reaching very competitive accuracy rates.

READ FULL TEXT

page 2

page 11

page 12

research
12/02/2019

Using Dimensionality Reduction to Optimize t-SNE

t-SNE is a popular tool for embedding multi-dimensional datasets into tw...
research
09/09/2022

Improving Model Training via Self-learned Label Representations

Modern neural network architectures have shown remarkable success in sev...
research
11/08/2016

Inferring low-dimensional microstructure representations using convolutional neural networks

We apply recent advances in machine learning and computer vision to a ce...
research
04/14/2014

Random forests with random projections of the output space for high dimensional multi-label classification

We adapt the idea of random projections applied to the output space, so ...
research
06/24/2020

Multilabel Classification by Hierarchical Partitioning and Data-dependent Grouping

In modern multilabel classification problems, each data instance belongs...
research
06/13/2021

Atlas Based Representation and Metric Learning on Manifolds

We explore the use of a topological manifold, represented as a collectio...

Please sign up or login with your details

Forgot password? Click here to reset