research
∙
06/30/2023
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit
In deep learning theory, the covariance matrix of the representations se...
share
research
∙
12/27/2022
Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization
To date, no "information-theoretic" frameworks for reasoning about gener...
research
∙
12/26/2022
Statistical minimax theorems via nonstandard analysis
For statistical decision problems with finite parameter space, it is wel...
research
∙
10/25/2022
Pruning's Effect on Generalization Through the Lens of Training and Regularization
Practitioners frequently observe that pruning improves model generalizat...
research
∙
07/25/2022
Statistical Inference with Stochastic Gradient Algorithms
Tuning of stochastic gradient algorithms (SGAs) for optimization and sam...
research
∙
06/29/2022
Understanding Generalization via Leave-One-Out Conditional Mutual Information
We study the mutual information between (certain summaries of) the outpu...
research
∙
06/06/2022
The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
The logit outputs of a feedforward neural network at initialization are ...
research
∙
02/10/2022
Adaptively Exploiting d-Separators with Causal Bandits
Multi-armed bandit problems provide a framework to identify the optimal ...
research
∙
12/28/2021
Admissibility is Bayes optimality with infinitesimals
We give an exact characterization of admissibility in statistical decisi...
research
∙
11/09/2021
Towards a Unified Information-Theoretic Framework for Generalization
In this work, we investigate the expressiveness of the "conditional mutu...
research
∙
09/21/2021
Minimax Rates for Conditional Density Estimation via Empirical Entropy
We consider the task of estimating a conditional density using i.i.d. sa...
research
∙
06/07/2021
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
Theoretical results show that neural networks can be approximated by Gau...
research
∙
04/28/2021
NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need...
research
∙
12/14/2020
NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
Understanding generalization in deep learning is arguably one of the mos...
research
∙
11/07/2020
Existence of matching priors on compact spaces
A matching prior at level 1-α is a prior such that an associated 1-α cre...
research
∙
11/05/2020
On the Information Complexity of Proper Learners for VC Classes in the Realizable Case
We provide a negative resolution to a conjecture of Steinke and Zakynthi...
research
∙
10/28/2020
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
In suitably initialized wide networks, small learning rates transform de...
research
∙
10/26/2020
Enforcing Interpretability and its Statistical Impacts: Trade-offs between Accuracy and Interpretability
To date, there has been no formal study of the statistical cost of inter...
research
∙
10/22/2020
In Search of Robust Measures of Generalization
One of the principal scientific challenges in deep learning is explainin...
research
∙
09/18/2020
Pruning Neural Networks at Initialization: Why are We Missing the Mark?
Recent work has explored the possibility of pruning neural networks at i...
research
∙
07/13/2020
Relaxing the I.I.D. Assumption: Adaptive Minimax Optimal Sequential Prediction with Expert Advice
We consider sequential prediction with expert advice when the data are g...
research
∙
07/02/2020
Improved Bounds on Minimax Regret under Logarithmic Loss via Self-Concordance
We consider the classical problem of sequential probability assignment u...
research
∙
06/19/2020
On the role of data in PAC-Bayes bounds
The dominant term in PAC-Bayes bounds is often the Kullback–Leibler dive...
research
∙
04/27/2020
Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
The information-theoretic framework of Russo and J. Zou (2016) and Xu an...
research
∙
12/14/2019
Approximations in Probabilistic Programs
We study the first-order probabilistic programming language introduced b...
research
∙
12/11/2019
Linear Mode Connectivity and the Lottery Ticket Hypothesis
We introduce "instability analysis," a framework for assessing whether t...
research
∙
12/09/2019
In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictors
We propose to study the generalization error of a learned predictor ĥ in...
research
∙
11/06/2019
Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
In this work, we improve upon the stepwise analysis of noisy iterative l...
research
∙
08/20/2019
Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processes
The developments of Rademacher complexity and PAC-Bayesian theory have b...
research
∙
08/17/2019
Black-box constructions for exchangeable sequences of random multisets
We develop constructions for exchangeable sequences of point processes t...
research
∙
08/16/2019
NUQSGD: Improved Communication Efficiency for Data-parallel SGD via Nonuniform Quantization
As the size and complexity of models and datasets grow, so does the need...
research
∙
03/05/2019
The Lottery Ticket Hypothesis at Scale
Recent work on the "lottery ticket hypothesis" proposes that randomly-in...
research
∙
02/26/2018
The Beta-Bernoulli process and algebraic effects
In this paper we analyze the Beta-Bernoulli process from Bayesian nonpar...
research
∙
02/26/2018
Data-dependent PAC-Bayes priors via differential privacy
The Probably Approximately Correct (PAC) Bayes framework (McAllester, 19...
research
∙
01/31/2018
On the computability of graphons
We investigate the relative computability of exchangeable binary relatio...
research
∙
12/26/2017
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy
We show that Entropy-SGD (Chaudhari et al., 2016), when viewed as a lear...
research
∙
12/06/2017
Exchangeable modelling of relational data: checking sparsity, train-test splitting, and sparse exchangeable Poisson matrix factorization
A variety of machine learning tasks---e.g., matrix factorization, topic ...
research
∙
12/05/2017
An estimator for the tail-index of graphex processes
Sparse exchangeable graphs resolve some pathologies in traditional rando...
research
∙
08/02/2016
A study of the effect of JPG compression on adversarial images
Neural network image classifiers are known to be vulnerable to adversari...
research
∙
07/07/2016
A characterization of product-form exchangeable feature probability functions
We characterize the class of exchangeable feature allocations assigning ...
research
∙
06/16/2016
The Mondrian Kernel
We introduce the Mondrian kernel, a fast random feature approximation to...
research
∙
06/07/2016
Measuring the reliability of MCMC inference with bidirectional Monte Carlo
Markov chain Monte Carlo (MCMC) is one of the main workhorses of probabi...
research
∙
12/08/2015
Gibbs-type Indian buffet processes
We investigate a class of feature allocation models that generalize the ...
research
∙
11/19/2015
Neural Network Matrix Factorization
Data often comes in the form of an array or matrix. Matrix factorization...
research
∙
06/11/2015
Mondrian Forests for Large-Scale Regression when Uncertainty Matters
Many real-world regression problems demand a measure of the uncertainty ...
research
∙
05/14/2015
Training generative neural networks via Maximum Mean Discrepancy optimization
We consider training a deep neural network to generate samples from an u...
research
∙
02/16/2015
Particle Gibbs for Bayesian Additive Regression Trees
Additive regression trees are flexible non-parametric models and popular...
research
∙
12/31/2014
The continuum-of-urns scheme, generalized beta and Indian buffet processes, and hierarchies thereof
We describe the combinatorial stochastic process underlying a sequence o...
research
∙
12/31/2013
The combinatorial structure of beta negative binomial processes
We characterize the combinatorial structure of conditionally-i.i.d. sequ...
research
∙
12/30/2013
