Information in Infinite Ensembles of Infinitely-Wide Neural Networks

11/20/2019

∙

In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.

READ FULL TEXT

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Information Flow in Deep Neural Networks

Bayesian Deep Ensembles via the Neural Tangent Kernel

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

Towards Further Understanding of Sparse Filtering via Information Bottleneck

Information Plane Analysis for Dropout Neural Networks

Limitations of the NTK for Understanding Generalization in Deep Learning

A Compositional Atlas of Tractable Circuit Operations: From Simple Transformations to Complex Information-Theoretic Queries

Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Related Research

Information Flow in Deep Neural Networks

Bayesian Deep Ensembles via the Neural Tangent Kernel

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

Towards Further Understanding of Sparse Filtering via Information Bottleneck

Information Plane Analysis for Dropout Neural Networks

Limitations of the NTK for Understanding Generalization in Deep Learning

A Compositional Atlas of Tractable Circuit Operations: From Simple Transformations to Complex Information-Theoretic Queries