Łukasz Kaiser

research

∙ 11/24/2021

Sparse is Enough in Scaling Transformers

Large Transformer models yield impressive results on many tasks, but are...

6 Sebastian Jaszczur, et al. ∙

research

∙ 10/26/2021

Hierarchical Transformers Are More Efficient Language Models

Transformer models yield impressive results on many NLP and sequence mod...

46 Piotr Nawrot, et al. ∙

research

∙ 07/07/2021

Evaluating Large Language Models Trained on Code

We introduce Codex, a GPT language model fine-tuned on publicly availabl...

6 Mark Chen, et al. ∙

research

∙ 02/12/2021

Q-Value Weighted Regression: Reinforcement Learning with Limited Data

Sample efficiency and performance in the offline setting have emerged as...

41 Piotr Kozakowski, et al. ∙

research

∙ 09/30/2020

Rethinking Attention with Performers

We introduce Performers, Transformer architectures which can estimate re...

5 Krzysztof Choromanski, et al. ∙

research

∙ 01/13/2020

Reformer: The Efficient Transformer

Large Transformer models routinely achieve state-of-the-art results on a...

13 Nikita Kitaev, et al. ∙

research

∙ 06/11/2019

Parallel Scheduled Sampling

Auto-regressive models are widely used in sequence generation problems. ...

0 Daniel Duckworth, et al. ∙

research

∙ 05/21/2019

Sample Efficient Text Summarization Using a Single Pre-Trained Transformer

Language model (LM) pre-training has resulted in impressive performance ...

0 Urvashi Khandelwal, et al. ∙

research

∙ 03/01/2019

Model-Based Reinforcement Learning for Atari

Model-free reinforcement learning (RL) can be used to learn effective po...

20 Łukasz Kaiser, et al. ∙

research

∙ 10/23/2018

Area Attention

Existing attention mechanisms, are mostly item-based in that a model is ...

4 Yang Li, et al. ∙

research

∙ 07/10/2018

Universal Transformers

Self-attentive feed-forward sequence models have been shown to achieve i...

6 Mostafa Dehghani, et al. ∙

research

∙ 03/16/2018

Tensor2Tensor for Neural Machine Translation

Tensor2Tensor is a library for deep learning models that is well-suited ...

0 Ashish Vaswani, et al. ∙

research

∙ 03/09/2018

Fast Decoding in Sequence Models using Discrete Latent Variables

Autoregressive sequence models based on deep neural networks, such as RN...

0 Łukasz Kaiser, et al. ∙

research

∙ 02/15/2018

Image Transformer

Image generation has been successfully cast as an autoregressive sequenc...

0 Niki Parmar, et al. ∙

research

∙ 02/15/2018

Image Tranformer

Image generation has been successfully cast as an autoregressive sequenc...

0 Niki Parmar, et al. ∙

research

∙ 01/30/2018

Generating Wikipedia by Summarizing Long Sequences

We show that generating English Wikipedia articles can be approached as ...

0 Peter J. Liu, et al. ∙

research

∙ 01/29/2018

Discrete Autoencoders for Sequence Models

Recurrent models for sequences have been recently successful at many tas...

0 Łukasz Kaiser, et al. ∙

research

∙ 01/15/2018

Unsupervised Cipher Cracking Using Discrete GANs

This work details CipherGAN, an architecture inspired by CycleGAN used f...

0 Aidan N. Gomez, et al. ∙

research

∙ 06/16/2017

One Model To Learn Them All

Deep learning yields great results across many fields, from speech recog...

0 Łukasz Kaiser, et al. ∙

research

∙ 06/12/2017

Attention Is All You Need

The dominant sequence transduction models are based on complex recurrent...

0 Ashish Vaswani, et al. ∙

research

∙ 06/09/2017

Depthwise Separable Convolutions for Neural Machine Translation

Depthwise separable convolutions reduce the number of parameters and com...

0 Łukasz Kaiser, et al. ∙

research

∙ 01/23/2017

Regularizing Neural Networks by Penalizing Confident Output Distributions

We systematically explore regularizing neural networks by penalizing low...

0 Gabriel Pereyra, et al. ∙

research

∙ 10/27/2016

Can Active Memory Replace Attention?

Several mechanisms to focus attention of a neural network on selected pa...

0 Łukasz Kaiser, et al. ∙

research

∙ 09/26/2016

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Neural Machine Translation (NMT) is an end-to-end learning approach for ...

0 Yonghui Wu, et al. ∙

research

∙ 03/14/2016

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

TensorFlow is an interface for expressing machine learning algorithms, a...

0 Martín Abadi, et al. ∙

research

∙ 11/25/2015

Neural GPUs Learn Algorithms

Learning an algorithm from examples is a fundamental problem that has be...

0 Łukasz Kaiser, et al. ∙

research

∙ 11/21/2015

Adding Gradient Noise Improves Learning for Very Deep Networks

Deep feedforward and recurrent networks have achieved impressive results...

0 Arvind Neelakantan, et al. ∙

research

∙ 11/19/2015

Multi-task Sequence to Sequence Learning

Sequence to sequence learning has recently emerged as a new paradigm in ...

0 Minh-Thang Luong, et al. ∙

research

∙ 12/23/2014

Grammar as a Foreign Language

Syntactic constituency parsing is a fundamental problem in natural langu...

0 Oriol Vinyals, et al. ∙

Łukasz Kaiser

Featured Co-authors