Beyond Human-Level Accuracy: Computational Challenges in Deep Learning

09/03/2019
by   Joel Hestness, et al.
0

Deep learning (DL) research yields accuracy and product improvements from both model architecture changes and scale: larger data sets and models, and more computation. For hardware design, it is difficult to predict DL model changes. However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably. This paper leverages the prior work to project the dataset and model size growth required to advance DL accuracy beyond human-level, to frontier targets defined by machine learning experts. Datasets will need to grow 33--971 ×, while models will need to grow 6.6--456× to achieve target accuracies. We further characterize and project the computational requirements to train these applications at scale. Our characterization reveals an important segmentation of DL training challenges for recurrent neural networks (RNNs) that contrasts with prior studies of deep convolutional networks. RNNs will have comparatively moderate operational intensities and very large memory footprint requirements. In contrast to emerging accelerator designs, large-scale RNN training characteristics suggest designs with significantly larger memory capacity and on-chip caches.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/01/2017

Deep Learning Scaling is Predictable, Empirically

Deep learning (DL) creates impactful advances following a virtuous recip...
research
02/27/2020

Optimizing Memory-Access Patterns for Deep Learning Accelerators

Deep learning (DL) workloads are moving towards accelerators for faster ...
research
10/08/2018

Deep Learning with the Random Neural Network and its Applications

The random neural network (RNN) is a mathematical model for an "integrat...
research
08/01/2023

Simulating the Geometric Growth of the Marine Sponge Crella Incrustans

Simulating marine sponge growth helps marine biologists analyze, measure...
research
01/03/2021

Silicon Photonic Microring Based Chip-Scale Accelerator for Delayed Feedback Reservoir Computing

To perform temporal and sequential machine learning tasks, the use of co...
research
04/17/2020

DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications

One of the limitations of deep learning models with sparse features toda...
research
06/27/2022

Efficient Deep Learning Using Non-Volatile Memory Technology

Embedded machine learning (ML) systems have now become the dominant plat...

Please sign up or login with your details

Forgot password? Click here to reset