Jascha Sohl-Dickstein

research

∙ 06/17/2022

Fast Finite Width Neural Tangent Kernel

The Neural Tangent Kernel (NTK), defined as Θ_θ^f(x_1, x_2) = [∂ f(θ, x_...

Roman Novak, et al. ∙

research

∙ 06/15/2022

Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling

We introduce repriorisation, a data-dependent reparameterisation which t...

Jiri Hron, et al. ∙

research

∙ 03/22/2022

Practical tradeoffs between memory, compute, and performance in learned optimizers

Optimization plays a costly and crucial role in developing machine learn...

Luke Metz, et al. ∙

research

∙ 12/27/2021

Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies

Unrolled computation graphs arise in many scenarios, including training ...

Paul Vicol, et al. ∙

research

∙ 01/14/2021

Training Learned Optimizers with Randomly Initialized Learned Optimizers

Learned optimizers are increasingly effective, with performance exceedin...

Luke Metz, et al. ∙

research

∙ 12/07/2020

Parallel Training of Deep Networks with Local Updates

Deep learning models trained on large data sets have been widely success...

Michael (Misha) Laskin, et al. ∙

research

∙ 11/11/2020

Towards NNGP-guided Neural Architecture Search

The predictions of wide Bayesian neural networks are described by a Gaus...

Daniel S. Park, et al. ∙

research

∙ 11/04/2020

Reverse engineering learned optimizers reveals known and novel mechanisms

Learned optimizers are algorithms that can themselves be trained to solv...

Niru Maheswaranathan, et al. ∙

research

∙ 10/21/2020

Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence

We perform an extensive empirical study of the statistical properties of...

Vinay Rao, et al. ∙

research

∙ 09/23/2020

Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

Much as replacing hand-designed features with learned functions has revo...

Luke Metz, et al. ∙

research

∙ 08/17/2020

Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible

Machine learning is predicated on the concept of generalization: a model...

Neha S. Wadia, et al. ∙

research

∙ 07/31/2020

Finite Versus Infinite Neural Networks: an Empirical Study

We perform a careful, thorough, and large scale empirical study of the c...

Jaehoon Lee, et al. ∙

research

∙ 07/17/2020

A new method for parameter estimation in probabilistic models: Minimum probability flow

Fitting probabilistic models to data is often difficult, due to the gene...

Jascha Sohl-Dickstein, et al. ∙

research

∙ 06/18/2020

Exact posterior distributions of wide Bayesian neural networks

Recent work has shown that the prior over functions induced by a deep Ba...

Jiri Hron, et al. ∙

research

∙ 06/18/2020

Infinite attention: NNGP and NTK for deep attention networks

There is a growing amount of literature on the relationship between wide...

Jiri Hron, et al. ∙

research

∙ 05/13/2020

Two equalities expressing the determinant of a matrix in terms of expectations over matrix-vector products

We introduce two equations expressing the inverse determinant of a full ...

Jascha Sohl-Dickstein, et al. ∙

research

∙ 03/12/2020

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

We show that the sum of the implicit generator log-density log p_g of a ...

Tong Che, et al. ∙

research

∙ 03/04/2020

The large learning rate phase of deep learning: the catapult mechanism

The choice of initial learning rate can have a profound effect on the pe...

Aitor Lewkowycz, et al. ∙

research

∙ 02/27/2020

Using a thousand optimization tasks to learn hyperparameter search strategies

We present TaskSet, a dataset of tasks for use in training and evaluatin...

Luke Metz, et al. ∙

research

∙ 01/21/2020

On the infinite width limit of neural networks with a standard parameterization

There are currently two parameterizations used to derive fixed kernels c...

Jascha Sohl-Dickstein, et al. ∙

research

∙ 12/05/2019

Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Neural Tangents is a library designed to enable research into infinite-w...

Roman Novak, et al. ∙

research

∙ 09/10/2019

Neural reparameterization improves structural optimization

Structural optimization is a popular method for designing objects such a...

Stephan Hoyer, et al. ∙

research

∙ 06/08/2019

Using learned optimizers to make models robust to input noise

State-of-the art vision models can achieve superhuman performance on ima...

Luke Metz, et al. ∙

research

∙ 05/09/2019

The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study

We investigate how the final parameters found by stochastic gradient des...

Daniel S. Park, et al. ∙

research

∙ 03/18/2019

A RAD approach to deep mixture models

Flow based models such as Real NVP are an extremely powerful approach to...

Laurent Dinh, et al. ∙

research

∙ 02/21/2019

A Mean Field Theory of Batch Normalization

We develop a mean field theory for batch normalization in fully-connecte...

Greg Yang, et al. ∙

research

∙ 02/18/2019

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

A longstanding goal in deep learning research has been to precisely char...

Jaehoon Lee, et al. ∙

research

∙ 01/12/2019

Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit

Recent work has noted that all bad local minima can be removed from neur...

Jascha Sohl-Dickstein, et al. ∙

research

∙ 11/08/2018

Measuring the Effects of Data Parallelism on Neural Network Training

Recent hardware developments have made unprecedented amounts of data par...

Christopher J. Shallue, et al. ∙

research

∙ 10/24/2018

Learned optimizers that outperform SGD on wall-clock and test loss

Deep learning has shown that learned functions can dramatically outperfo...

Luke Metz, et al. ∙

research

∙ 10/24/2018

Learned optimizers that outperform SGD on wall-clock and validation loss

Deep learning has shown that learned functions can dramatically outperfo...

Luke Metz, et al. ∙

research

∙ 10/11/2018

Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes

There is a previously identified equivalence between wide fully connecte...

Roman Novak, et al. ∙

research

∙ 06/28/2018

Adversarial Reprogramming of Neural Networks

Deep neural networks are susceptible to adversarial attacks. In computer...

Gamaleldin F. Elsayed, et al. ∙

research

∙ 06/26/2018

Guided evolutionary strategies: escaping the curse of dimensionality in random search

Many applications in machine learning require optimizing a function whos...

Niru Maheswaranathan, et al. ∙

research

∙ 06/25/2018

Stochastic natural gradient descent draws posterior samples in function space

Natural gradient descent (NGD) minimises the cost function on a Riemanni...

Samuel L. Smith, et al. ∙

research

∙ 06/22/2018

PCA of high dimensional random walks with comparison to neural network training

One technique to visualize the training of neural networks is to perform...

Joseph M. Antognini, et al. ∙

research

∙ 06/14/2018

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

In recent years, state-of-the-art methods in computer vision have utiliz...

Lechao Xiao, et al. ∙

research

∙ 03/31/2018

Learning Unsupervised Learning Rules

A major goal of unsupervised learning is to discover data representation...

Luke Metz, et al. ∙

research

∙ 02/23/2018

Sensitivity and Generalization in Neural Networks: an Empirical Study

In practice it is often found that large over-parameterized neural netwo...

Roman Novak, et al. ∙

research

∙ 02/22/2018

Adversarial Examples that Fool both Human and Computer Vision

Machine learning models are vulnerable to adversarial examples: small ch...

Gamaleldin F. Elsayed, et al. ∙

research

∙ 11/25/2017

Generalizing Hamiltonian Monte Carlo with Neural Networks

We present a general-purpose method to train Markov chain Monte Carlo ke...

Daniel Lévy, et al. ∙

research

∙ 11/01/2017

Deep Neural Networks as Gaussian Processes

A deep fully-connected neural network with an i.i.d. prior over its para...

Jaehoon Lee, et al. ∙

research

∙ 10/18/2017

A Correspondence Between Random Neural Networks and Statistical Field Theory

A number of recent papers have provided evidence that practical design q...

Samuel S. Schoenholz, et al. ∙

research

∙ 06/19/2017

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

We propose a new technique, Singular Vector Canonical Correlation Analys...

Maithra Raghu, et al. ∙

research

∙ 03/21/2017

REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

Learning in models with discrete latent variables is challenging due to ...

George Tucker, et al. ∙

research

∙ 03/14/2017

Learned Optimizers that Scale and Generalize

Learning to learn has emerged as an important direction for achieving ar...

Olga Wichrowska, et al. ∙

research

∙ 11/29/2016

Capacity and Trainability in Recurrent Neural Networks

Two potential bottlenecks on the expressiveness of recurrent neural netw...

Jasmine Collins, et al. ∙

research

∙ 11/28/2016

Input Switched Affine Networks: An RNN Architecture Designed for Interpretability

There exist many problem domains where the interpretability of neural ne...

Jakob N. Foerster, et al. ∙

research

∙ 11/24/2016

Survey of Expressivity in Deep Neural Networks

We survey results on neural network expressivity described in "On the Ex...

Maithra Raghu, et al. ∙

research

∙ 11/07/2016

Unrolled Generative Adversarial Networks

We introduce a method to stabilize Generative Adversarial Networks (GANs...

Luke Metz, et al. ∙

Jascha Sohl-Dickstein

Featured Co-authors