research
∙
08/21/2023
Unlocking Accuracy and Fairness in Differentially Private Image Classification
Privacy-preserving machine learning aims to train models on private data...
share
research
∙
07/21/2023
On the Universality of Linear Recurrences Followed by Nonlinear Projections
In this note (work in progress towards a full-length paper) we show that...
research
∙
03/11/2023
Resurrecting Recurrent Neural Networks for Long Sequences
Recurrent Neural Networks (RNNs) offer fast inference on long sequences ...
research
∙
02/27/2023
Differentially Private Diffusion Models Generate Useful Synthetic Images
The ability to generate privacy-preserving synthetic versions of sensiti...
research
∙
02/20/2023
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation
Skip connections and normalisation layers form two standard architectura...
research
∙
04/28/2022
Unlocking High-Accuracy Differentially Private Image Classification through Scale
Differential Privacy (DP) provides a formal privacy guarantee preventing...
research
∙
05/27/2021
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
In computer vision, it is standard practice to draw a single sample from...
research
∙
02/11/2021
High-Performance Large-Scale Image Recognition Without Normalization
Batch normalization is a key component of most image classification mode...
research
∙
01/28/2021
On the Origin of Implicit Regularization in Stochastic Gradient Descent
For infinitesimal learning rates, stochastic gradient descent (SGD) foll...
research
∙
01/21/2021
Characterizing signal propagation to close the performance gap in unnormalized ResNets
Batch Normalization is a key component in almost all state-of-the-art im...
research
∙
07/31/2020
Cold Posteriors and Aleatoric Uncertainty
Recent work has observed that one can outperform exact inference in Baye...
research
∙
06/26/2020
On the Generalization Benefit of Noise in Stochastic Gradient Descent
It has long been argued that minibatch stochastic gradient descent can g...
research
∙
02/24/2020
Batch Normalization Biases Deep Residual Networks Towards Shallow Paths
Batch normalization has multiple benefits. It improves the conditioning ...
research
∙
05/09/2019
The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
We investigate how the final parameters found by stochastic gradient des...
research
∙
06/25/2018
Stochastic natural gradient descent draws posterior samples in function space
Natural gradient descent (NGD) minimises the cost function on a Riemanni...
research
∙
05/09/2018
Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Experimental evidence indicates that simple models outperform complex de...
research
∙
11/01/2017
Don't Decay the Learning Rate, Increase the Batch Size
It is common practice to decay the learning rate. Here we show one can u...
research
∙
10/17/2017
A Bayesian Perspective on Generalization and Stochastic Gradient Descent
This paper tackles two related questions at the heart of machine learnin...
research
∙
02/13/2017
Offline bilingual word vectors, orthogonal transformations and the inverted softmax
Usually bilingual word vectors are trained "online". Mikolov et al. show...
research
∙
12/27/2016