DeepAI AI Chat
Log In Sign Up

Complement Objective Training

by   Hao-Yun Chen, et al.

Learning with a primary objective, such as softmax cross entropy for classification and sequence generation, has been the norm for training deep neural networks for years. Although being a widely-adopted approach, using cross entropy as the primary objective exploits mostly the information from the ground-truth class for maximizing data likelihood, and largely ignores information from the complement (incorrect) classes. We argue that, in addition to the primary objective, training also using a complement objective that leverages information from the complement classes can be effective in improving model performance. This motivates us to study a new training paradigm that maximizes the likelihood of the groundtruth class while neutralizing the probabilities of the complement classes. We conduct extensive experiments on multiple tasks ranging from computer vision to natural language understanding. The experimental results confirm that, compared to the conventional training with just one primary objective, training also with the complement objective further improves the performance of the state-of-the-art models across all tasks. In addition to the accuracy improvement, we also show that models trained with both primary and complement objectives are more robust to single-step adversarial attacks.


page 1

page 2

page 3

page 4


Improving Adversarial Robustness via Guided Complement Entropy

Model robustness has been an important issue, since adding small adversa...

Imbalanced Image Classification with Complement Cross Entropy

Recently, deep learning models have achieved great success in computer v...

Learning with Hierarchical Complement Objective

Label hierarchies widely exist in many vision-related problems, ranging ...

Improving Document Image Understanding with Reinforcement Finetuning

Successful Artificial Intelligence systems often require numerous labele...

SCL: Towards Accurate Domain Adaptive Object Detection via Gradient Detach Based Stacked Complementary Losses

Unsupervised domain adaptive object detection aims to learn a robust det...

Differentiable N-gram Objective on Abstractive Summarization

ROUGE is a standard automatic evaluation metric based on n-grams for seq...

Cosine similarity-based adversarial process

An adversarial process between two deep neural networks is a promising a...