Jascha Sohl-Dickstein

is this you? claim profile

∙ 0 followers
Google ∙ Stanford University

Staff Research Scientist in the Brain group at Google, Academic Resident at Khan Academy, Visiting scholar in Surya Ganguli's lab at Stanford University, PhD in 2012 in the Redwood Center for Theoretical Neuroscience at UC Berkeley, in Bruno Olshausen's lab.

Featured Co-authors

Yoshua Bengio
352 publications
Pieter Abbeel
174 publications
Quoc V. Le
92 publications
Nando de Freitas
73 publications
Hugo Larochelle
66 publications
Leonidas Guibas
65 publications
Razvan Pascanu
62 publications
Yuan Cao
50 publications
Ian Goodfellow
50 publications
Jon Kleinberg
48 publications
Samy Bengio
48 publications

Training Learned Optimizers with Randomly Initialized Learned Optimizers

Learned optimizers are increasingly effective, with performance exceedin...
01/14/2021 ∙ by Luke Metz, et al. ∙ 8 ∙ share

read it
Parallel Training of Deep Networks with Local Updates

Deep learning models trained on large data sets have been widely success...
12/07/2020 ∙ by Michael (Misha) Laskin, et al. ∙ 8 ∙ share

read it
Towards NNGP-guided Neural Architecture Search

The predictions of wide Bayesian neural networks are described by a Gaus...
11/11/2020 ∙ by Daniel S. Park, et al. ∙ 6 ∙ share

read it
Reverse engineering learned optimizers reveals known and novel mechanisms

Learned optimizers are algorithms that can themselves be trained to solv...
11/04/2020 ∙ by Niru Maheswaranathan, et al. ∙ 22 ∙ share

read it
Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence

We perform an extensive empirical study of the statistical properties of...
10/21/2020 ∙ by Vinay Rao, et al. ∙ 0 ∙ share

read it
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

Much as replacing hand-designed features with learned functions has revo...
09/23/2020 ∙ by Luke Metz, et al. ∙ 12 ∙ share

read it
Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible

Machine learning is predicated on the concept of generalization: a model...
08/17/2020 ∙ by Neha S. Wadia, et al. ∙ 0 ∙ share

read it
Finite Versus Infinite Neural Networks: an Empirical Study

We perform a careful, thorough, and large scale empirical study of the c...
07/31/2020 ∙ by Jaehoon Lee, et al. ∙ 49 ∙ share

read it
A new method for parameter estimation in probabilistic models: Minimum probability flow

Fitting probabilistic models to data is often difficult, due to the gene...
07/17/2020 ∙ by Jascha Sohl-Dickstein, et al. ∙ 0 ∙ share

read it
Exact posterior distributions of wide Bayesian neural networks

Recent work has shown that the prior over functions induced by a deep Ba...
06/18/2020 ∙ by Jiri Hron, et al. ∙ 0 ∙ share

read it
Infinite attention: NNGP and NTK for deep attention networks

There is a growing amount of literature on the relationship between wide...
06/18/2020 ∙ by Jiri Hron, et al. ∙ 0 ∙ share

read it
Two equalities expressing the determinant of a matrix in terms of expectations over matrix-vector products

We introduce two equations expressing the inverse determinant of a full ...
05/13/2020 ∙ by Jascha Sohl-Dickstein, et al. ∙ 0 ∙ share

read it
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

We show that the sum of the implicit generator log-density log p_g of a ...
03/12/2020 ∙ by Tong Che, et al. ∙ 24 ∙ share

read it
The large learning rate phase of deep learning: the catapult mechanism

The choice of initial learning rate can have a profound effect on the pe...
03/04/2020 ∙ by Aitor Lewkowycz, et al. ∙ 0 ∙ share

read it
Using a thousand optimization tasks to learn hyperparameter search strategies

We present TaskSet, a dataset of tasks for use in training and evaluatin...
02/27/2020 ∙ by Luke Metz, et al. ∙ 71 ∙ share

read it
On the infinite width limit of neural networks with a standard parameterization

There are currently two parameterizations used to derive fixed kernels c...
01/21/2020 ∙ by Jascha Sohl-Dickstein, et al. ∙ 13 ∙ share

read it
Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Neural Tangents is a library designed to enable research into infinite-w...
12/05/2019 ∙ by Roman Novak, et al. ∙ 14 ∙ share

read it
Neural reparameterization improves structural optimization

Structural optimization is a popular method for designing objects such a...
09/10/2019 ∙ by Stephan Hoyer, et al. ∙ 0 ∙ share

read it
Using learned optimizers to make models robust to input noise

State-of-the art vision models can achieve superhuman performance on ima...
06/08/2019 ∙ by Luke Metz, et al. ∙ 0 ∙ share

read it
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study

We investigate how the final parameters found by stochastic gradient des...
05/09/2019 ∙ by Daniel S. Park, et al. ∙ 0 ∙ share

read it
A RAD approach to deep mixture models

Flow based models such as Real NVP are an extremely powerful approach to...
03/18/2019 ∙ by Laurent Dinh, et al. ∙ 4 ∙ share

read it
A Mean Field Theory of Batch Normalization

We develop a mean field theory for batch normalization in fully-connecte...
02/21/2019 ∙ by Greg Yang, et al. ∙ 0 ∙ share

read it
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

A longstanding goal in deep learning research has been to precisely char...
02/18/2019 ∙ by Jaehoon Lee, et al. ∙ 0 ∙ share

read it
Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit

Recent work has noted that all bad local minima can be removed from neur...
01/12/2019 ∙ by Jascha Sohl-Dickstein, et al. ∙ 0 ∙ share

read it
Measuring the Effects of Data Parallelism on Neural Network Training

Recent hardware developments have made unprecedented amounts of data par...
11/08/2018 ∙ by Christopher J. Shallue, et al. ∙ 0 ∙ share

read it
Learned optimizers that outperform SGD on wall-clock and test loss

Deep learning has shown that learned functions can dramatically outperfo...
10/24/2018 ∙ by Luke Metz, et al. ∙ 8 ∙ share

read it
Learned optimizers that outperform SGD on wall-clock and validation loss

Deep learning has shown that learned functions can dramatically outperfo...
10/24/2018 ∙ by Luke Metz, et al. ∙ 2 ∙ share

read it
Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes

There is a previously identified equivalence between wide fully connecte...
10/11/2018 ∙ by Roman Novak, et al. ∙ 18 ∙ share

read it
Adversarial Reprogramming of Neural Networks

Deep neural networks are susceptible to adversarial attacks. In computer...
06/28/2018 ∙ by Gamaleldin F. Elsayed, et al. ∙ 10 ∙ share

read it
Guided evolutionary strategies: escaping the curse of dimensionality in random search

Many applications in machine learning require optimizing a function whos...
06/26/2018 ∙ by Niru Maheswaranathan, et al. ∙ 2 ∙ share

read it
Stochastic natural gradient descent draws posterior samples in function space

Natural gradient descent (NGD) minimises the cost function on a Riemanni...
06/25/2018 ∙ by Samuel L. Smith, et al. ∙ 0 ∙ share

read it
PCA of high dimensional random walks with comparison to neural network training

One technique to visualize the training of neural networks is to perform...
06/22/2018 ∙ by Joseph M. Antognini, et al. ∙ 0 ∙ share

read it
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

In recent years, state-of-the-art methods in computer vision have utiliz...
06/14/2018 ∙ by Lechao Xiao, et al. ∙ 0 ∙ share

read it
Learning Unsupervised Learning Rules

A major goal of unsupervised learning is to discover data representation...
03/31/2018 ∙ by Luke Metz, et al. ∙ 0 ∙ share

read it
Sensitivity and Generalization in Neural Networks: an Empirical Study

In practice it is often found that large over-parameterized neural netwo...
02/23/2018 ∙ by Roman Novak, et al. ∙ 0 ∙ share

read it
Adversarial Examples that Fool both Human and Computer Vision

Machine learning models are vulnerable to adversarial examples: small ch...
02/22/2018 ∙ by Gamaleldin F. Elsayed, et al. ∙ 0 ∙ share

read it
Generalizing Hamiltonian Monte Carlo with Neural Networks

We present a general-purpose method to train Markov chain Monte Carlo ke...
11/25/2017 ∙ by Daniel Lévy, et al. ∙ 0 ∙ share

read it
Deep Neural Networks as Gaussian Processes

A deep fully-connected neural network with an i.i.d. prior over its para...
11/01/2017 ∙ by Jaehoon Lee, et al. ∙ 0 ∙ share

read it
A Correspondence Between Random Neural Networks and Statistical Field Theory

A number of recent papers have provided evidence that practical design q...
10/18/2017 ∙ by Samuel S. Schoenholz, et al. ∙ 0 ∙ share

read it
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

We propose a new technique, Singular Vector Canonical Correlation Analys...
06/19/2017 ∙ by Maithra Raghu, et al. ∙ 0 ∙ share

read it
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

Learning in models with discrete latent variables is challenging due to ...
03/21/2017 ∙ by George Tucker, et al. ∙ 0 ∙ share

read it
Learned Optimizers that Scale and Generalize

Learning to learn has emerged as an important direction for achieving ar...
03/14/2017 ∙ by Olga Wichrowska, et al. ∙ 0 ∙ share

read it
Capacity and Trainability in Recurrent Neural Networks

Two potential bottlenecks on the expressiveness of recurrent neural netw...
11/29/2016 ∙ by Jasmine Collins, et al. ∙ 0 ∙ share

read it
Input Switched Affine Networks: An RNN Architecture Designed for Interpretability

There exist many problem domains where the interpretability of neural ne...
11/28/2016 ∙ by Jakob N. Foerster, et al. ∙ 0 ∙ share

read it
Survey of Expressivity in Deep Neural Networks

We survey results on neural network expressivity described in "On the Ex...
11/24/2016 ∙ by Maithra Raghu, et al. ∙ 0 ∙ share

read it
Unrolled Generative Adversarial Networks

We introduce a method to stabilize Generative Adversarial Networks (GANs...
11/07/2016 ∙ by Luke Metz, et al. ∙ 0 ∙ share

read it
Deep Information Propagation

We study the behavior of untrained neural networks whose weights and bia...
11/04/2016 ∙ by Samuel S. Schoenholz, et al. ∙ 0 ∙ share

read it
Exponential expressivity in deep neural networks through transient chaos

We combine Riemannian geometry with the mean field theory of high dimens...
06/16/2016 ∙ by Ben Poole, et al. ∙ 0 ∙ share

read it
On the Expressive Power of Deep Neural Networks

We propose a new approach to the problem of neural network expressivity,...
06/16/2016 ∙ by Maithra Raghu, et al. ∙ 0 ∙ share

read it
Density estimation using Real NVP

Unsupervised learning of probabilistic models is a central yet challengi...
05/27/2016 ∙ by Laurent Dinh, et al. ∙ 0 ∙ share

read it