b'Erich Elsen'

research

∙ 06/17/2022

The State of Sparse Training in Deep Reinforcement Learning

The use of sparse neural networks has seen rapid growth in recent years,...

Laura Graesser, et al. ∙

research

∙ 03/29/2022

Training Compute-Optimal Large Language Models

We investigate the optimal model size and number of tokens for training ...

Jordan Hoffmann, et al. ∙

research

∙ 02/02/2022

Unified Scaling Laws for Routed Language Models

The performance of a language model has been shown to be effectively mod...

Aidan Clark, et al. ∙

research

∙ 12/13/2021

Step-unrolled Denoising Autoencoders for Text Generation

In this paper we propose a new generative model of text, Step-unrolled D...

Nikolay Savinov, et al. ∙

research

∙ 12/08/2021

Improving language models by retrieving from trillions of tokens

We enhance auto-regressive language models by conditioning on document c...

Sebastian Borgeaud, et al. ∙

research

∙ 06/07/2021

Top-KAST: Top-K Always Sparse Training

Sparse neural networks are becoming increasingly important as the field ...

Siddhant M. Jayakumar, et al. ∙

research

∙ 06/26/2020

On the Generalization Benefit of Noise in Stochastic Gradient Descent

It has long been argued that minibatch stochastic gradient descent can g...

Samuel L. Smith, et al. ∙

research

∙ 06/18/2020

Sparse GPU Kernels for Deep Learning

Scientific workloads have traditionally exploited high levels of sparsit...

Trevor Gale, et al. ∙

research

∙ 06/12/2020

AlgebraNets

Neural networks have historically been built layerwise from the set of f...

Jordan Hoffmann, et al. ∙

research

∙ 06/12/2020

A Practical Sparse Approximation for Real Time Recurrent Learning

Current methods for training recurrent neural networks are based on back...

Jacob Menick, et al. ∙

research

∙ 06/05/2020

End-to-End Adversarial Text-to-Speech

Modern text-to-speech synthesis pipelines typically involve multiple pro...

Jeff Donahue, et al. ∙

research

∙ 11/25/2019

Rigging the Lottery: Making All Tickets Winners

Sparse neural networks have been shown to be more parameter and compute ...

Utku Evci, et al. ∙

research

∙ 11/21/2019

Fast Sparse ConvNets

Historically, the pursuit of efficient inference has been one of the dri...

Erich Elsen, et al. ∙

research

∙ 09/25/2019

High Fidelity Speech Synthesis with Adversarial Networks

Generative adversarial networks have seen rapid development in recent ye...

Mikołaj Bińkowski, et al. ∙

research

∙ 06/25/2019

The Difficulty of Training Sparse Neural Networks

We investigate the difficulties of training sparse neural networks and m...

Utku Evci, et al. ∙

research

∙ 06/07/2019

Non-Differentiable Supervised Learning with Evolution Strategies and Hybrid Methods

In this work we show that Evolution Strategies (ES) are a viable method ...

Karel Lenc, et al. ∙

research

∙ 02/25/2019

The State of Sparsity in Deep Neural Networks

We rigorously evaluate three state-of-the-art techniques for inducing sp...

Trevor Gale, et al. ∙

research

∙ 10/29/2018

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

Generating musical audio directly with neural networks is notoriously di...

Curtis Hawthorne, et al. ∙

research

∙ 02/23/2018

Efficient Neural Audio Synthesis

Sequential models achieve state-of-the-art results in audio, visual and ...

Nal Kalchbrenner, et al. ∙

research

∙ 11/28/2017

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

The recently-developed WaveNet architecture is the current state of the ...

Aaron van den Oord, et al. ∙

research

∙ 10/30/2017

Onsets and Frames: Dual-Objective Piano Transcription

We consider the problem of transcribing polyphonic piano music with an e...

Curtis Hawthorne, et al. ∙

research

∙ 10/10/2017

Mixed Precision Training

Deep neural networks have enabled progress in a wide variety of applicat...

Paulius Micikevicius, et al. ∙

research

∙ 04/17/2017

Exploring Sparsity in Recurrent Neural Networks

Recurrent Neural Networks (RNN) are widely used to solve a variety of pr...

Sharan Narang, et al. ∙

research

∙ 07/15/2016

DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Modern deep neural networks have a large number of parameters, making th...

Song Han, et al. ∙

research

∙ 12/08/2015

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

We show that an end-to-end deep learning approach can be used to recogni...

Dario Amodei, et al. ∙

research

∙ 12/17/2014

Deep Speech: Scaling up end-to-end speech recognition

We present a state-of-the-art speech recognition system developed using ...

Awni Hannun, et al. ∙

Erich Elsen

Featured Co-authors

Hao Wu

Yi Wang

Oriol Vinyals

Razvan Pascanu

Song Han

Chong Wang

Karen Simonyan

Bryan Catanzaro

Matei Zaharia

Andrew Y. Ng

Colin Raffel

The State of Sparse Training in Deep Reinforcement Learning

Training Compute-Optimal Large Language Models

Unified Scaling Laws for Routed Language Models

Step-unrolled Denoising Autoencoders for Text Generation

Improving language models by retrieving from trillions of tokens

Top-KAST: Top-K Always Sparse Training

On the Generalization Benefit of Noise in Stochastic Gradient Descent

Sparse GPU Kernels for Deep Learning

AlgebraNets

A Practical Sparse Approximation for Real Time Recurrent Learning

End-to-End Adversarial Text-to-Speech

Rigging the Lottery: Making All Tickets Winners

Fast Sparse ConvNets

High Fidelity Speech Synthesis with Adversarial Networks

The Difficulty of Training Sparse Neural Networks

Non-Differentiable Supervised Learning with Evolution Strategies and Hybrid Methods

The State of Sparsity in Deep Neural Networks

Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

Efficient Neural Audio Synthesis

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Onsets and Frames: Dual-Objective Piano Transcription

Mixed Precision Training

Exploring Sparsity in Recurrent Neural Networks

DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Deep Speech: Scaling up end-to-end speech recognition