b'Noam Shazeer'

research

∙ 04/05/2022

PaLM: Scaling Language Modeling with Pathways

Large language models have been shown to achieve remarkable performance ...

6 Aakanksha Chowdhery, et al. ∙

research

∙ 03/31/2022

Scaling Up Models and Data with and

Recent neural network-based language models have benefited greatly from ...

8 Adam Roberts, et al. ∙

research

∙ 02/17/2022

Designing Effective Sparse Expert Models

Scale has opened new frontiers in natural language processing – but at a...

0 Barret Zoph, et al. ∙

research

∙ 09/17/2021

Primer: Searching for Efficient Transformers for Language Modeling

Large Transformer models have been central to recent advances in natural...

0 David R. So, et al. ∙

research

∙ 05/10/2021

GSPMD: General and Scalable Parallelization for ML Computation Graphs

We present GSPMD, an automatic, compiler-based parallelization system fo...

4 Yuanzhong Xu, et al. ∙

research

∙ 02/23/2021

Do Transformer Modifications Transfer Across Implementations and Applications?

The research community has proposed copious modifications to the Transfo...

10 Sharan Narang, et al. ∙

research

∙ 01/11/2021

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

In deep learning, models typically reuse the same parameters for all inp...

0 William Fedus, et al. ∙

research

∙ 06/30/2020

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Neural network scaling has been critical for improving the model quality...

7 Dmitry Lepikhin, et al. ∙

research

∙ 03/05/2020

Talking-Heads Attention

We introduce "talking-heads attention" - a variation on multi-head atten...

0 Noam Shazeer, et al. ∙

research

∙ 02/12/2020

GLU Variants Improve Transformer

Gated Linear Units (arXiv:1612.08083) consist of the component-wise prod...

0 Noam Shazeer, et al. ∙

research

∙ 02/10/2020

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

It has recently been observed that neural language models trained on uns...

0 Adam Roberts, et al. ∙

research

∙ 01/14/2020

Faster Transformer Decoding: N-gram Masked Self-Attention

Motivated by the fact that most of the information relevant to the predi...

0 Ciprian Chelba, et al. ∙

research

∙ 11/06/2019

Fast Transformer Decoding: One Write-Head is All You Need

Multi-head attention layers, as used in the Transformer neural sequence ...

0 Noam Shazeer, et al. ∙

research

∙ 10/23/2019

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Transfer learning, where a model is first pre-trained on a data-rich tas...

0 Colin Raffel, et al. ∙

research

∙ 09/06/2019

High Resolution Medical Image Analysis with Spatial Partitioning

Medical images such as 3D computerized tomography (CT) scans and patholo...

0 Le Hou, et al. ∙

research

∙ 04/10/2019

Corpora Generation for Grammatical Error Correction

Grammatical Error Correction (GEC) has been recently modeled using the s...

0 Jared Lichtarge, et al. ∙

research

∙ 11/07/2018

Blockwise Parallel Decoding for Deep Autoregressive Models

Deep autoregressive sequence-to-sequence models have demonstrated impres...

0 Mitchell Stern, et al. ∙

research

∙ 11/05/2018

Mesh-TensorFlow: Deep Learning for Supercomputers

Batch-splitting (data-parallelism) is the dominant distributed Deep Neur...

8 Noam Shazeer, et al. ∙

research

∙ 10/31/2018

Weakly Supervised Grammatical Error Correction using Iterative Decoding

We describe an approach to Grammatical Error Correction (GEC) that is ef...

0 Jared Lichtarge, et al. ∙

research

∙ 09/12/2018

An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation

Music relies heavily on self-reference to build structure and meaning. W...

2 Cheng-Zhi Anna Huang, et al. ∙

research

∙ 09/12/2018

Music Transformer

Music relies heavily on repetition to build structure and meaning. Self-...

0 Cheng-Zhi Anna Huang, et al. ∙

research

∙ 04/11/2018

Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

In several recently proposed stochastic optimization methods (e.g. RMSPr...

0 Noam Shazeer, et al. ∙

research

∙ 03/16/2018

Tensor2Tensor for Neural Machine Translation

Tensor2Tensor is a library for deep learning models that is well-suited ...

0 Ashish Vaswani, et al. ∙

research

∙ 03/09/2018

Fast Decoding in Sequence Models using Discrete Latent Variables

Autoregressive sequence models based on deep neural networks, such as RN...

0 Łukasz Kaiser, et al. ∙

research

∙ 02/15/2018

Image Transformer

Image generation has been successfully cast as an autoregressive sequenc...

0 Niki Parmar, et al. ∙

research

∙ 02/15/2018

Image Tranformer

Image generation has been successfully cast as an autoregressive sequenc...

0 Niki Parmar, et al. ∙

research

∙ 01/30/2018

Generating Wikipedia by Summarizing Long Sequences

We show that generating English Wikipedia articles can be approached as ...

0 Peter J. Liu, et al. ∙

research

∙ 06/16/2017

One Model To Learn Them All

Deep learning yields great results across many fields, from speech recog...

0 Łukasz Kaiser, et al. ∙

research

∙ 06/12/2017

Attention Is All You Need

The dominant sequence transduction models are based on complex recurrent...

0 Ashish Vaswani, et al. ∙

research

∙ 01/23/2017

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

The capacity of a neural network to absorb information is limited by its...

0 Noam Shazeer, et al. ∙

research

∙ 06/23/2016

NN-grams: Unifying neural network and n-gram language models for Speech Recognition

We present NN-grams, a novel, hybrid language model integrating n-grams ...

0 Babak Damavandi, et al. ∙

research

∙ 02/07/2016

Exploring the Limits of Language Modeling

In this work we explore recent advances in Recurrent Neural Networks for...

0 Rafal Jozefowicz, et al. ∙

research

∙ 02/06/2016

Swivel: Improving Embeddings by Noticing What's Missing

We present Submatrix-wise Vector Embedding Learner (Swivel), a method fo...

0 Noam Shazeer, et al. ∙

research

∙ 06/09/2015

Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

Recurrent Neural Networks can be trained to produce sequences of tokens ...

0 Samy Bengio, et al. ∙

research

∙ 12/03/2014

Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation

We present a novel family of language model (LM) estimation techniques n...

0 Noam Shazeer, et al. ∙

research

∙ 06/04/2010

Variational Program Inference

We introduce a framework for representing a variety of interesting probl...

0 Georges Harik, et al. ∙

Noam Shazeer

Featured Co-authors