A Data-Centric Optimization Framework for Machine Learning

by   Oliver Rausch, et al.

Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute. However, as frameworks specialize optimization to patterns in popular networks, they implicitly constrain novel and diverse models that drive progress in research. We empower deep learning researchers by defining a flexible and user-customizable pipeline for optimizing training of arbitrary deep neural networks, based on data movement minimization. The pipeline begins with standard networks in PyTorch or ONNX and transforms computation through progressive lowering. We define four levels of general-purpose transformations, from local intra-operator optimizations to global data movement reduction. These operate on a data-centric graph intermediate representation that expresses computation and data movement at all levels of abstraction, including expanding basic operators such as convolutions to their underlying computations. Central to the design is the interactive and introspectable nature of the pipeline. Every part is extensible through a Python API, and can be tuned interactively using a GUI. We demonstrate competitive performance or speedups on ten different networks, with interactive optimizations discovering new opportunities in EfficientNet.



page 3

page 8

page 10

page 16


Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads

Recent trend towards increasing large machine learning models require bo...

RVSDG: An Intermediate Representation for Optimizing Compilers

Intermediate Representations (IRs) are central to optimizing compilers a...

PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python

Machine learning is a general-purpose technology holding promises for ma...

Weld: Rethinking the Interface Between Data-Intensive Applications

Data analytics applications combine multiple functions from different li...

Stateful Dataflow Multigraphs: A Data-Centric Model for Performance Portability on Heterogeneous Architectures

The ubiquity of accelerators in high-performance computing has driven pr...

Data Movement Is All You Need: A Case Study on Optimizing Transformers

Transformers have become widely used for language modeling and sequence ...

Guiding Optimizations with Meliora: A Deep Walk down Memory Lane

Performance models can be very useful for understanding the behavior of ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.