Regularising Deep Networks with DGMs

09/25/2019
by   Matthew Willetts, et al.
10

Here we develop a new method for regularising neural networks where we learn a density estimator over the activations of all layers of the model. We extend recent work in data imputation using VAEs (Ivanov et al., 2018) so that we can obtain a posterior for an arbitrary subset of activations conditioned on the remainder. Our method has links both to dropout and to data augmentation. We demonstrate that our training method leads to lower cross-entropy test set loss for 2-hidden-layer neural networks trained on CIFAR-10 and SVHN compared to standard regularisation baselines, but our model does not improve test-set accuracy compared to our baselines. This implies that although decisions are broadly similar, our approach provides a network with better calibrated uncertainty measures over the class posteriors.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/12/2022

Examining the Proximity of Adversarial Examples to Class Manifolds in Deep Networks

Deep neural networks achieve remarkable performance in multiple fields. ...
research
08/02/2018

Likelihood-free inference with an improved cross-entropy estimator

We extend recent work (Brehmer, et. al., 2018) that use neural networks ...
research
11/17/2022

Data-Centric Debugging: mitigating model failures via targeted data collection

Deep neural networks can be unreliable in the real world when the traini...
research
07/18/2022

Interpolation, extrapolation, and local generalization in common neural networks

There has been a long history of works showing that neural networks have...
research
12/14/2019

Bayesian Linear Regression on Deep Representations

A simple approach to obtaining uncertainty-aware neural networks for reg...
research
04/04/2023

Calibrated Chaos: Variance Between Runs of Neural Network Training is Harmless and Inevitable

Typical neural network trainings have substantial variance in test-set p...
research
12/09/2021

PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures

In real-world applications of machine learning, reliable and safe system...

Please sign up or login with your details

Forgot password? Click here to reset