research
∙
05/13/2023
Depth Dependence of μP Learning Rates in ReLU MLPs
In this short note we consider random fully connected ReLU networks of w...
share
research
∙
01/30/2023
On student-teacher deviations in distillation: does it pay to disobey?
Knowledge distillation has been widely-used to improve the performance o...
research
∙
10/19/2022
On the Adversarial Robustness of Mixture of Experts
Adversarial robustness is a key desirable property of neural networks. I...
research
∙
10/12/2022
Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers
This paper studies the curious phenomenon for machine learning models wi...
research
∙
02/02/2022
Robust Training of Neural Networks using Scale Invariant Architectures
In contrast to SGD, adaptive gradient methods like Adam allow robust tra...
research
∙
06/19/2021
Teacher's pet: understanding and mitigating biases in distillation
Knowledge distillation is widely used as a means of improving the perfor...
research
∙
04/18/2021
Demystifying the Better Performance of Position Encoding Variants for Transformer
Transformers are state of the art models in NLP that map a given input s...
research
∙
03/26/2021
Understanding Robustness of Transformers for Image Classification
Deep Convolutional Neural Networks (CNNs) have long been the architectur...
research
∙
02/05/2021
On the Reproducibility of Neural Network Predictions
Standard training techniques for neural networks involve multiple source...
research
∙
12/01/2020
Modifying Memories in Transformer Models
Large Transformer models have achieved impressive performance in many na...
research
∙
10/27/2020
An efficient nonconvex reformulation of stagewise convex optimization problems
Convex optimization problems with staged structure appear in several con...
research
∙
10/23/2020
Coping with Label Shift via Distributionally Robust Optimisation
The label shift problem refers to the supervised learning setting where ...
research
∙
06/08/2020
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
Transformer networks use pairwise attention to compute contextual embedd...
research
∙
03/05/2020
Does label smoothing mitigate label noise?
Label smoothing is commonly used in training deep learning models, where...
research
∙
02/17/2020
Low-Rank Bottleneck in Multi-head Attention Models
Attention based Transformer architecture has enabled significant advance...
research
∙
12/20/2019
Are Transformers universal approximators of sequence-to-sequence functions?
Despite the widespread adoption of Transformer models for NLP tasks, the...
research
∙
05/30/2018
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
Despite existing work on ensuring generalization of neural networks in t...
research
∙
03/01/2018
Smoothed analysis for low-rank solutions to semidefinite programs in quadratic penalty form
Semidefinite programs (SDP) are important in learning and combinatorial ...
research
∙
11/04/2017
Provable quantum state tomography via non-convex methods
With nowadays steadily growing quantum processors, it is required to dev...
research
∙
05/25/2017
Implicit Regularization in Matrix Factorization
We study implicit regularization when optimizing an underdetermined quad...
research
∙
05/22/2017
Stabilizing GAN Training with Multiple Random Projections
Training generative adversarial networks is unstable in high-dimensions ...
research
∙
10/21/2016
Single Pass PCA of Matrix Products
In this paper we present a new algorithm for computing a low rank approx...
research
∙
06/04/2016
Provable Burer-Monteiro factorization for a class of norm-constrained matrix problems
We study the projected gradient descent method on low-rank matrix proble...
research
∙
05/23/2016
Global Optimality of Local Search for Low Rank Matrix Recovery
We show that there are no spurious local minima in the non-convex factor...
research
∙
09/14/2015
Dropping Convexity for Faster Semi-definite Optimization
We study the minimization of a convex function f(X) over the set of n× n...
research
∙
02/17/2015
A New Sampling Technique for Tensors
In this paper we propose new techniques to sample arbitrary third-order ...
research
∙
10/14/2014
Tighter Low-rank Approximation via Sampling the Leveraged Element
In this work, we propose a new randomized algorithm for computing a low-...
research
∙
02/10/2014
Universal Matrix Completion
The problem of low-rank matrix completion has recently generated a lot o...
research
∙
06/12/2013