DeepAI AI Chat
Log In Sign Up

Variance Networks: When Expectation Does Not Meet Your Expectations

03/10/2018
by   Kirill Neklyudov, et al.
0

In this paper, we propose variance networks, a new model that stores the learned information in the variances of the network weights. Surprisingly, no information gets stored in the expectations of the weights, therefore if we replace these weights with their expectations, we would obtain a random guess quality prediction. We provide a numerical criterion that uses the loss curvature to determine which random variables can be replaced with their expected values, and find that only a small fraction of weights is needed for ensembling. Variance networks represent a diverse ensemble that is more robust to adversarial attacks than conventional low-variance ensembles. The success of this model raises several counter-intuitive implications for the training and application of Deep Learning models.

READ FULL TEXT

page 1

page 2

page 3

page 4

09/30/2022

Learning Robust Kernel Ensembles with Kernel Average Pooling

Model ensembles have long been used in machine learning to reduce the va...
09/27/2021

MUTEN: Boosting Gradient-Based Adversarial Attacks via Mutant-Based Ensembles

Deep Neural Networks (DNNs) are vulnerable to adversarial examples, whic...
04/01/2019

Stratified Random Sampling for Dependent Inputs

A new approach of obtaining stratified random samples from statistically...
11/09/2022

REDS: Random Ensemble Deep Spatial prediction

There has been a great deal of recent interest in the development of spa...
05/24/2020

Sharp variance-entropy comparison for nonnegative gaussian quadratic forms

In this article we study quadratic forms in n independent standard norma...
05/04/2019

ARSM: Augment-REINFORCE-Swap-Merge Estimator for Gradient Backpropagation Through Categorical Variables

To address the challenge of backpropagating the gradient through categor...
02/10/2020

Introducing Graph Cumulants: What is the Variance of Your Social Network?

In an increasingly interconnected world, understanding and summarizing t...