research
∙
06/30/2023
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
We study the type of solutions to which stochastic gradient descent conv...
share
research
∙
06/18/2023
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Background: Distributed training is essential for large scale training o...
research
∙
06/06/2023
Continual Learning in Linear Classification on Separable Data
We analyze continual learning on a sequence of separable linear classifi...
research
∙
06/05/2023
Explore to Generalize in Zero-Shot RL
We study zero-shot generalization in reinforcement learning - optimizing...
research
∙
05/22/2023
Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond
Recent research shows that when Gradient Descent (GD) is applied to neur...
research
∙
03/14/2023
Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations
Although CNNs are believed to be invariant to translations, recent works...
research
∙
02/10/2023
The Role of Codeword-to-Class Assignments in Error-Correcting Codes: An Empirical Study
Error-correcting codes (ECC) are used to reduce multiclass classificatio...
research
∙
05/19/2022
How catastrophic can catastrophic forgetting be in linear regression?
To better understand catastrophic forgetting, we study fitting an overpa...
research
∙
03/21/2022
Optimal Fine-Grained N:M sparsity for Activations and Neural Gradients
In deep learning, fine-grained N:M sparsity reduces the data footprint a...
research
∙
12/19/2021
Logarithmic Unbiased Quantization: Practical 4-bit Training in Deep Learning
Quantization of the weights and activations is one of the main methods t...
research
∙
09/24/2021
Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability
In the Bayesian reinforcement learning (RL) setting, a prior distributio...
research
∙
06/14/2021
Physics-Aware Downsampling with Deep Learning for Scalable Flood Modeling
Background: Floods are the most common natural disaster in the world, af...
research
∙
02/25/2021
Statistical Testing for Efficient Out of Distribution Detection in Deep Neural Networks
Commonly, Deep Neural Networks (DNNs) generalize well on samples drawn f...
research
∙
02/19/2021
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
Recent work has highlighted the role of initialization scale in determin...
research
∙
02/16/2021
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
Recently, researchers proposed pruning deep neural network weights (DNNs...
research
∙
10/01/2020
Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
Background: Catastrophic forgetting is the notorious vulnerability of ne...
research
∙
07/13/2020
Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
We provide a detailed asymptotic study of gradient flow trajectories and...
research
∙
07/02/2020
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?
Deep neural networks are typically initialized with random weights, with...
research
∙
06/15/2020
Neural gradients are lognormally distributed: understanding sparse and quantized training
Neural gradient compression remains a main bottleneck in improving train...
research
∙
02/20/2020
Kernel and Rich Regimes in Overparametrized Models
A recent line of work studies overparametrized neural networks in the "k...
research
∙
12/29/2019
MTJ-Based Hardware Synapse Design for Quantized Deep Neural Networks
Quantized neural networks (QNNs) are being actively researched as a solu...
research
∙
12/11/2019
Is Feature Diversity Necessary in Neural Network Initialization?
Standard practice in training neural networks involves initializing the ...
research
∙
12/03/2019
The Knowledge Within: Methods for Data-Free Model Compression
Background: Recently, an extensive amount of research has been focused o...
research
∙
10/03/2019
A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
A key element of understanding the efficacy of overparameterized neural ...
research
∙
09/26/2019
At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
Background: Recent developments have made it possible to accelerate neur...
research
∙
06/13/2019
Kernel and Deep Regimes in Overparametrized Models
A recent line of work studies overparametrized neural networks in the "k...
research
∙
06/03/2019
A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
Reducing the precision of weights and activation functions in neural net...
research
∙
05/17/2019
Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
With an eye toward understanding complexity control in deep learning, we...
research
∙
02/13/2019
How do infinite width bounded norm networks look in function space?
We consider the question of what functions can be captured by ReLU netwo...
research
∙
01/27/2019
Augment your batch: better training with larger batches
Large-batch SGD is important for scaling training of deep neural network...
research
∙
10/02/2018
ACIQ: Analytical Clipping for Integer Quantization of neural networks
Unlike traditional approaches that focus on the quantization at the netw...
research
∙
06/05/2018
Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
Stochastic Gradient Descent (SGD) is a central tool in machine learning....
research
∙
06/01/2018
Implicit Bias of Gradient Descent on Linear Convolutional Networks
We show that gradient descent on full-width linear convolutional network...
research
∙
05/25/2018
Scalable Methods for 8-bit Training of Neural Networks
Quantized Neural Networks (QNNs) are often used to improve network effic...
research
∙
05/13/2018
The Global Optimization Geometry of Shallow Linear Neural Networks
We examine the squared error loss landscape of shallow linear neural net...
research
∙
03/27/2018
Bayesian Gradient Descent: Online Variational Bayes Learning with Increased Robustness to Catastrophic Forgetting and Weight Pruning
We suggest a novel approach for the estimation of the posterior distribu...
research
∙
03/05/2018
Convergence of Gradient Descent on Separable Data
The implicit bias of gradient descent is not fully understood even in si...
research
∙
03/05/2018
Norm matters: efficient and accurate normalization schemes in deep networks
Over the past few years batch-normalization has been commonly used in de...
research
∙
02/22/2018
Characterizing Implicit Bias in Terms of Optimization Geometry
We study the bias of generic optimization methods, including Mirror Desc...
research
∙
02/14/2018
On the Blindspots of Convolutional Networks
Deep convolutional network has been the state-of-the-art approach for a ...
research
∙
01/14/2018
Fix your classifier: the marginal value of training the last weight layer
Neural networks are commonly used as models for classification for a wid...
research
∙
10/27/2017
The Implicit Bias of Gradient Descent on Separable Data
We show that gradient descent on an unregularized logistic regression pr...
research
∙
05/24/2017
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Background: Deep learning models are typically trained using stochastic ...
research
∙
02/19/2017
Exponentially vanishing sub-optimal local minima in multilayer neural networks
Background: Statistical mechanics results (Dauphin et al. (2014); Chorom...
research
∙
09/22/2016
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
We introduce a method to train Quantized Neural Networks (QNNs) --- neur...
research
∙
05/26/2016
No bad local minima: Data independent training error guarantees for multilayer neural networks
We use smoothed analysis techniques to provide guarantees on the trainin...
research
∙
02/08/2016
Binarized Neural Networks
We introduce a method to train Binarized Neural Networks (BNNs) - neural...
research
∙
03/12/2015
Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation
Compared to Multilayer Neural Networks with real weights, Binary Multila...
research
∙
10/07/2013