research
∙
08/23/2023
D4: Improving LLM Pretraining via Document De-Duplication and Diversification
Over recent years, an increasing amount of compute and data has been pou...
share
research
∙
08/08/2023
PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning
Synthetic image datasets offer unmatched advantages for designing and ev...
research
∙
05/27/2023
On the special role of class-selective neurons in early training
It is commonly observed that deep networks trained for classification ex...
research
∙
03/16/2023
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Progress in machine learning has been driven in large part by massive in...
research
∙
01/30/2023
Emergence of Maps in the Memories of Blind Navigation Agents
Animal navigation research posits that organisms build and maintain inte...
research
∙
10/19/2022
lo-fi: distributed fine-tuning without communication
When fine-tuning large neural networks, it is common to use multiple nod...
research
∙
06/29/2022
Beyond neural scaling laws: beating power law scaling via data pruning
Widely observed neural scaling laws, in which error falls off as a power...
research
∙
03/10/2022
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
The conventional recipe for maximizing model accuracy is to (1) train mu...
research
∙
06/09/2021
Grounding inductive biases in natural images:invariance stems from variations in data
To perform well on unseen and potentially out-of-distribution samples, i...
research
∙
04/24/2021
Width Transfer: On the (In)variance of Width Optimization
Optimizing the channel counts for different layers of a CNN has shown gr...
research
∙
03/23/2021
Leveraging background augmentations to encourage semantic focus in self-supervised contrastive learning
Unsupervised representation learning is an important challenge in comput...
research
∙
12/30/2020
Reservoir Transformer
We demonstrate that transformers obtain impressive performance even when...
research
∙
10/13/2020
Are all negatives created equal in contrastive instance discrimination?
Self-supervised learning has recently begun to rival supervised learning...
research
∙
07/23/2020
PareCO: Pareto-aware Channel Optimization for Slimmable Neural Networks
Slimmable neural networks provide a flexible trade-off front between pre...
research
∙
07/08/2020
On the relationship between class selectivity, dimensionality, and robustness
While the relative trade-offs between sparse and distributed representat...
research
∙
05/07/2020
Plan2Vec: Unsupervised Representation Learning by Latent Plans
In this paper we introduce plan2vec, an unsupervised representation lear...
research
∙
02/29/2020
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Batch normalization (BatchNorm) has become an indispensable tool for tra...
research
∙
02/24/2020
The Early Phase of Neural Network Training
Recent studies have shown that many important aspects of neural network ...
research
∙
06/09/2019
The Generalization-Stability Tradeoff in Neural Network Pruning
Pruning neural network parameters to reduce model size is an area of muc...
research
∙
06/06/2019
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
The success of lottery ticket initializations (Frankle and Carbin, 2019)...
research
∙
06/06/2019
Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
The lottery ticket hypothesis proposes that over-parameterization of dee...
research
∙
01/31/2019
Learning to Make Analogies by Contrasting Abstract Relational Structure
Analogical reasoning has been a principal focus of various waves of AI r...
research
∙
10/31/2018
Analyzing biological and artificial neural networks: challenges with opportunities for synergy?
Deep neural networks (DNNs) transform stimuli across multiple processing...
research
∙
07/11/2018
Measuring abstract reasoning in neural networks
Whether neural networks can learn abstract reasoning or whether they mer...
research
∙
07/03/2018
Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
Recent progress in artificial intelligence through reinforcement learnin...
research
∙
06/14/2018
Insights on representational similarity in neural networks with canonical correlation
Comparing different neural network representations and determining how r...
research
∙
04/12/2018
Learned Deformation Stability in Convolutional Neural Networks
Conventional wisdom holds that interleaved pooling layers in convolution...
research
∙
03/19/2018