DeepAI AI Chat
Log In Sign Up

Deep Learning on Small Datasets without Pre-Training using Cosine Loss

by   Björn Barz, et al.

Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss function provides significantly better performance than cross-entropy on datasets with only a handful of samples per class. For example, the accuracy achieved on the CUB-200-2011 dataset without pre-training is by 30 other popular datasets confirm our findings. Moreover, we show that the classification performance can be improved further by integrating prior knowledge in the form of class hierarchies, which is straightforward with the cosine loss.


Taming the Cross Entropy Loss

We present the Tamed Cross Entropy (TCE) loss function, a robust derivat...

LiteDepthwiseNet: An Extreme Lightweight Network for Hyperspectral Image Classification

Deep learning methods have shown considerable potential for hyperspectra...

Deformable CNN and Imbalance-Aware Feature Learning for Singing Technique Classification

Singing techniques are used for expressive vocal performances by employi...

Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning

Modern deep learning is primarily an experimental science, in which empi...

Multiscale Softmax Cross Entropy for Fovea Localization on Color Fundus Photography

Fovea localization is one of the most popular tasks in ophthalmic medica...

Leveraging Class Similarity to Improve Deep Neural Network Robustness

Traditionally artificial neural networks (ANNs) are trained by minimizin...