research
∙
05/26/2023
Randomized Positional Encodings Boost Length Generalization of Transformers
Transformers have impressive generalization capabilities on tasks with a...
share
research
∙
02/15/2023
Topological Neural Discrete Representation Learning à la Kohonen
Unsupervised learning of discrete representations from continuous ones i...
research
∙
10/12/2022
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations
Well-designed diagnostic tasks have played a key role in studying the fa...
research
∙
09/22/2022
A Generalist Neural Algorithmic Learner
The cornerstone of neural algorithmic reasoning is the ability to solve ...
research
∙
02/11/2022
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
Linear layers in neural networks (NNs) trained by gradient descent can b...
research
∙
02/11/2022
A Modern Self-Referential Weight Matrix That Learns to Modify Itself
The weight matrix (WM) of a neural network (NN) is its program. The prog...
research
∙
12/31/2021
Improving Baselines in the Wild
We share our experience with the recently released WILDS benchmark, a co...
research
∙
10/14/2021
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
Despite successes across a broad range of applications, Transformers hav...
research
∙
08/26/2021
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
Recently, many datasets have been proposed to test the systematic genera...
research
∙
06/11/2021
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Transformers with linearised attention ("linear Transformers") have demo...
research
∙
10/05/2020
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
Neural networks (NNs) whose subnetworks implement reusable functions are...
research
∙
04/23/2019