State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations

by   Alex Lamb, et al.

Machine learning promises methods that generalize well from finite labeled data. However, the brittleness of existing neural net approaches is revealed by notable failures, such as the existence of adversarial examples that are misclassified despite being nearly identical to a training example, or the inability of recurrent sequence-processing nets to stay on track without teacher forcing. We introduce a method, which we refer to as state reification, that involves modeling the distribution of hidden states over the training data and then projecting hidden states observed during testing toward this distribution. Our intuition is that if the network can remain in a familiar manifold of hidden space, subsequent layers of the net should be well trained to respond appropriately. We show that this state-reification method helps neural nets to generalize better, especially when labeled data are sparse, and also helps overcome the challenge of achieving robust generalization with adversarial training.



There are no comments yet.


page 1

page 2

page 3

page 4


Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations

Deep networks have achieved impressive results across a variety of impor...

Manifold Mixup: Encouraging Meaningful On-Manifold Interpolation as a Regularizer

Deep networks often perform well on the data manifold on which they are ...

From Deep to Shallow: Transformations of Deep Rectifier Networks

In this paper, we introduce transformations of deep rectifier networks, ...

The EOS Decision and Length Extrapolation

Extrapolation to unseen sequence lengths is a challenge for neural gener...

Adversarial Examples Detection and Analysis with Layer-wise Autoencoders

We present a mechanism for detecting adversarial examples based on data ...

Measuring Neural Net Robustness with Constraints

Despite having high accuracy, neural nets have been shown to be suscepti...

A+D-Net: Shadow Detection with Adversarial Shadow Attenuation

Single image shadow detection is a very challenging problem because of t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.