research
∙
04/05/2022
PaLM: Scaling Language Modeling with Pathways
Large language models have been shown to achieve remarkable performance ...
share
research
∙
03/31/2022
Scaling Up Models and Data with and
Recent neural network-based language models have benefited greatly from ...
research
∙
02/17/2022
Designing Effective Sparse Expert Models
Scale has opened new frontiers in natural language processing – but at a...
research
∙
09/17/2021
Primer: Searching for Efficient Transformers for Language Modeling
Large Transformer models have been central to recent advances in natural...
research
∙
05/10/2021
GSPMD: General and Scalable Parallelization for ML Computation Graphs
We present GSPMD, an automatic, compiler-based parallelization system fo...
research
∙
02/23/2021
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transfo...
research
∙
01/11/2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
In deep learning, models typically reuse the same parameters for all inp...
research
∙
06/30/2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Neural network scaling has been critical for improving the model quality...
research
∙
03/05/2020
Talking-Heads Attention
We introduce "talking-heads attention" - a variation on multi-head atten...
research
∙
02/12/2020
GLU Variants Improve Transformer
Gated Linear Units (arXiv:1612.08083) consist of the component-wise prod...
research
∙
02/10/2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model?
It has recently been observed that neural language models trained on uns...
research
∙
01/14/2020
Faster Transformer Decoding: N-gram Masked Self-Attention
Motivated by the fact that most of the information relevant to the predi...
research
∙
11/06/2019
Fast Transformer Decoding: One Write-Head is All You Need
Multi-head attention layers, as used in the Transformer neural sequence ...
research
∙
10/23/2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich tas...
research
∙
09/06/2019
High Resolution Medical Image Analysis with Spatial Partitioning
Medical images such as 3D computerized tomography (CT) scans and patholo...
research
∙
04/10/2019
Corpora Generation for Grammatical Error Correction
Grammatical Error Correction (GEC) has been recently modeled using the s...
research
∙
11/07/2018
Blockwise Parallel Decoding for Deep Autoregressive Models
Deep autoregressive sequence-to-sequence models have demonstrated impres...
research
∙
11/05/2018
Mesh-TensorFlow: Deep Learning for Supercomputers
Batch-splitting (data-parallelism) is the dominant distributed Deep Neur...
research
∙
10/31/2018
Weakly Supervised Grammatical Error Correction using Iterative Decoding
We describe an approach to Grammatical Error Correction (GEC) that is ef...
research
∙
09/12/2018
An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation
Music relies heavily on self-reference to build structure and meaning. W...
research
∙
09/12/2018
Music Transformer
Music relies heavily on repetition to build structure and meaning. Self-...
research
∙
04/11/2018
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
In several recently proposed stochastic optimization methods (e.g. RMSPr...
research
∙
03/16/2018
Tensor2Tensor for Neural Machine Translation
Tensor2Tensor is a library for deep learning models that is well-suited ...
research
∙
03/09/2018
Fast Decoding in Sequence Models using Discrete Latent Variables
Autoregressive sequence models based on deep neural networks, such as RN...
research
∙
02/15/2018
Image Transformer
Image generation has been successfully cast as an autoregressive sequenc...
research
∙
02/15/2018
Image Tranformer
Image generation has been successfully cast as an autoregressive sequenc...
research
∙
01/30/2018
Generating Wikipedia by Summarizing Long Sequences
We show that generating English Wikipedia articles can be approached as ...
research
∙
06/16/2017
One Model To Learn Them All
Deep learning yields great results across many fields, from speech recog...
research
∙
06/12/2017
Attention Is All You Need
The dominant sequence transduction models are based on complex recurrent...
research
∙
01/23/2017
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
The capacity of a neural network to absorb information is limited by its...
research
∙
06/23/2016
NN-grams: Unifying neural network and n-gram language models for Speech Recognition
We present NN-grams, a novel, hybrid language model integrating n-grams ...
research
∙
02/07/2016
Exploring the Limits of Language Modeling
In this work we explore recent advances in Recurrent Neural Networks for...
research
∙
02/06/2016
Swivel: Improving Embeddings by Noticing What's Missing
We present Submatrix-wise Vector Embedding Learner (Swivel), a method fo...
research
∙
06/09/2015
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Recurrent Neural Networks can be trained to produce sequences of tokens ...
research
∙
12/03/2014
Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation
We present a novel family of language model (LM) estimation techniques n...
research
∙
06/04/2010