research
∙
06/26/2023
Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression
Pretrained transformers exhibit the remarkable ability of in-context lea...
share
research
∙
06/07/2023
Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks
In this work, we reveal a strong implicit bias of stochastic gradient de...
research
∙
03/16/2023
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Progress in machine learning has been driven in large part by massive in...
research
∙
11/16/2022
Holistic Evaluation of Language Models
Language models (LMs) are becoming the foundation for almost all major l...
research
∙
10/15/2022
Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution
Neuroscience has long been an important driver of progress in artificial...
research
∙
10/11/2022
What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundaries
Deep neural network classifiers partition input space into high confiden...
research
∙
10/07/2022
The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks
In this work, we explore the maximum-margin bias of quasi-homogeneous ne...
research
∙
10/06/2022
Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?
Modern deep learning involves training costly, highly overparameterized ...
research
∙
09/30/2022
Disentangling with Biological Constraints: A Theory of Functional Cell Types
Neurons in the brain are often finely tuned for specific task variables....
research
∙
06/29/2022
Beyond neural scaling laws: beating power law scaling via data pruning
Widely observed neural scaling laws, in which error falls off as a power...
research
∙
06/02/2022
Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
A striking observation about iterative magnitude pruning (IMP; Frankle e...
research
∙
03/22/2022
MetaMorph: Learning Universal Controllers with Transformers
Multiple domains like vision, natural language, and audio are witnessing...
research
∙
07/19/2021
Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations, and anomalous diffusion
In this work we explore the limiting dynamics of deep neural networks tr...
research
∙
07/15/2021
Deep Learning on a Data Diet: Finding Important Examples Early in Training
The recent success of deep learning has partially been driven by trainin...
research
∙
07/13/2021
How many degrees of freedom do we need to train deep networks: a loss landscape perspective
A variety of recent works, spanning pruning, lottery tickets, and traini...
research
∙
02/12/2021
Understanding self-supervised Learning Dynamics without Contrastive Pairs
Contrastive approaches to self-supervised learning (SSL) learn represent...
research
∙
02/03/2021
Embodied Intelligence via Learning and Evolution
The intertwined processes of learning and evolution in complex environme...
research
∙
12/08/2020
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
Predicting the dynamics of neural network parameters during training is ...
research
∙
10/28/2020
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
In suitably initialized wide networks, small learning rates transform de...
research
∙
10/22/2020
Identifying Learning Rules From Neural Network Observables
The brain modifies its synaptic strengths during learning in order to be...
research
∙
10/15/2020
RNNs can generate bounded hierarchical languages with optimal memory
Recurrent neural networks empirically generate natural language with hig...
research
∙
10/01/2020
Understanding Self-supervised Learning with Dual Deep Networks
We propose a novel theoretical framework to understand self-supervised l...
research
∙
06/25/2020
Predictive coding in balanced neural networks with noise, chaos and delays
Biological neural networks face a formidable task: performing reliable c...
research
∙
06/09/2020
Pruning neural networks without any data by iteratively conserving synaptic flow
Pruning the parameters of deep neural networks has generated intense int...
research
∙
02/28/2020
Two Routes to Scalable Credit Assignment without Weight Symmetry
The neural plausibility of backpropagation has long been disputed, prima...
research
∙
12/12/2019
From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction
Recently, deep feedforward neural networks have achieved considerable su...
research
∙
10/14/2019
Emergent properties of the local geometry of neural loss landscapes
The local geometry of high dimensional neural network loss landscapes ca...
research
∙
07/19/2019
Universality and individuality in neural dynamics across large populations of recurrent networks
Task-based modeling with recurrent neural networks (RNNs) has emerged as...
research
∙
06/29/2019
Fast Convolutive Nonnegative Matrix Factorization Through Coordinate and Block Coordinate Updates
Identifying recurring patterns in high-dimensional time series data is a...
research
∙
06/25/2019
Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics
Recurrent neural networks (RNNs) are a widely used tool for modeling seq...
research
∙
01/03/2019
A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs
The visual system is hierarchically organized to process visual informat...
research
∙
10/23/2018
A mathematical theory of semantic development in deep neural networks
An extensive body of empirical research has revealed remarkable regulari...
research
∙
10/23/2018
Statistical mechanics of low-rank tensor decomposition
Often, large, high dimensional datasets collected across multiple modali...
research
∙
09/27/2018
An analytic theory of generalization dynamics and transfer learning in deep linear networks
Much attention has been devoted recently to the generalization puzzle in...
research
∙
06/20/2018
Task-Driven Convolutional Recurrent Models of the Visual System
Feed-forward convolutional neural networks (CNNs) are currently state-of...
research
∙
02/27/2018
The Emergence of Spectral Universality in Deep Networks
Recent work has shown that tight concentration of the entire spectrum of...
research
∙
11/13/2017
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
It is well known that the initialization of weights in deep neural netwo...
research
∙
11/07/2017
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
We propose a novel method to directly learn a stochastic transition oper...
research
∙
05/31/2017
SuperSpike: Supervised learning in multi-layer spiking neural networks
A vast majority of computation in the brain is performed by spiking neur...
research
∙
03/27/2017
Biologically inspired protection of deep networks from adversarial attacks
Inspired by biophysical principles underlying nonlinear dendritic comput...
research
∙
03/13/2017
Continual Learning Through Synaptic Intelligence
While deep learning has led to remarkable advances across diverse applic...
research
∙
02/06/2017
Deep Learning Models of the Retinal Response to Natural Scenes
A central challenge in neuroscience is to understand neural computations...
research
∙
11/24/2016
Survey of Expressivity in Deep Neural Networks
We survey results on neural network expressivity described in "On the Ex...
research
∙
11/04/2016
Deep Information Propagation
We study the behavior of untrained neural networks whose weights and bia...
research
∙
09/22/2016
An equivalence between high dimensional Bayes optimal inference and M-estimation
When recovering an unknown signal from noisy measurements, the computati...
research
∙
07/14/2016
Random projections of random manifolds
Interesting data often concentrate on low dimensional smooth manifolds i...
research
∙
06/16/2016
Exponential expressivity in deep neural networks through transient chaos
We combine Riemannian geometry with the mean field theory of high dimens...
research
∙
06/16/2016
On the Expressive Power of Deep Neural Networks
We propose a new approach to the problem of neural network expressivity,...
research
∙
03/24/2016
A universal tradeoff between power, precision and speed in physical communication
Maximizing the speed and precision of communication while minimizing pow...
research
∙
01/18/2016