research
∙
10/11/2022
What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries
Deep neural network classifiers partition input space into high confiden...
share
research
∙
07/19/2022
Deep equilibrium networks are sensitive to initialization statistics
Deep equilibrium networks (DEQs) are a promising way to construct models...
research
∙
06/17/2022
Fast Finite Width Neural Tangent Kernel
The Neural Tangent Kernel (NTK), defined as Θ_θ^f(x_1, x_2) = [∂ f(θ, x_...
research
∙
11/10/2021
Gradients are Not All You Need
Differentiable programming techniques are widely used in the community a...
research
∙
02/07/2021
Tilting the playing field: Dynamical loss functions for machine learning
We show that learning can be improved by using loss functions that evolv...
research
∙
08/17/2020
Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible
Machine learning is predicated on the concept of generalization: a model...
research
∙
07/31/2020
Finite Versus Infinite Neural Networks: an Empirical Study
We perform a careful, thorough, and large scale empirical study of the c...
research
∙
01/21/2020
On the infinite width limit of neural networks with a standard parameterization
There are currently two parameterizations used to derive fixed kernels c...
research
∙
12/30/2019
Disentangling trainability and generalization in deep learning
A fundamental goal in deep learning is the characterization of trainabil...
research
∙
12/09/2019
JAX, M.D.: End-to-End Differentiable, Hardware Accelerated, Molecular Dynamics in Pure Python
A large fraction of computational science involves simulating the dynami...
research
∙
12/05/2019
Neural Tangents: Fast and Easy Infinite Neural Networks in Python
Neural Tangents is a library designed to enable research into infinite-w...
research
∙
02/21/2019
A Mean Field Theory of Batch Normalization
We develop a mean field theory for batch normalization in fully-connecte...
research
∙
02/18/2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
A longstanding goal in deep learning research has been to precisely char...
research
∙
01/25/2019
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
Training recurrent neural networks (RNNs) on long sequence tasks is plag...
research
∙
08/20/2018
Peptide-Spectra Matching from Weak Supervision
As in many other scientific domains, we face a fundamental problem when ...
research
∙
06/14/2018
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
Recurrent neural networks have gained widespread use in modeling sequenc...
research
∙
06/14/2018
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
In recent years, state-of-the-art methods in computer vision have utiliz...
research
∙
02/27/2018
The Emergence of Spectral Universality in Deep Networks
Recent work has shown that tight concentration of the entire spectrum of...
research
∙
01/09/2018
Adversarial Spheres
State of the art computer vision models have been shown to be vulnerable...
research
∙
12/24/2017
Mean Field Residual Networks: On the Edge of Chaos
We study randomly initialized residual networks using mean field theory ...
research
∙
11/13/2017
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
It is well known that the initialization of weights in deep neural netwo...
research
∙
11/08/2017
Intriguing Properties of Adversarial Examples
It is becoming increasingly clear that many machine learning classifiers...
research
∙
11/01/2017
Deep Neural Networks as Gaussian Processes
A deep fully-connected neural network with an i.i.d. prior over its para...
research
∙
10/18/2017
A Correspondence Between Random Neural Networks and Statistical Field Theory
A number of recent papers have provided evidence that practical design q...
research
∙
09/23/2017
Combining Machine Learning and Physics to Understand Glassy Systems
Our understanding of supercooled liquids and glasses has lagged signific...
research
∙
11/04/2016