Equality Saturation for Tensor Graph Superoptimization

01/05/2021
by   Yichen Yang, et al.
0

One of the major optimizations employed in deep learning frameworks is graph rewriting. Production frameworks rely on heuristics to decide if rewrite rules should be applied and in which order. Prior research has shown that one can discover more optimal tensor computation graphs if we search for a better sequence of substitutions instead of relying on heuristics. However, we observe that existing approaches for tensor graph superoptimization both in production and research frameworks apply substitutions in a sequential manner. Such sequential search methods are sensitive to the order in which the substitutions are applied and often only explore a small fragment of the exponential space of equivalent graphs. This paper presents a novel technique for tensor graph superoptimization that employs equality saturation to apply all possible substitutions at once. We show that our approach can find optimized graphs with up to 16 time optimizing.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/19/2020

SPORES: Sum-Product Optimization via Relational Equality Saturation for Large Scale Linear Algebra

Machine learning algorithms are commonly specified in linear algebra (LA...
08/23/2021

Rewrite Rule Inference Using Equality Saturation

Many compilers, synthesizers, and theorem provers rely on rewrite rules ...
03/20/2020

New heuristics for burning graphs

The concept of graph burning and burning number (bn(G)) of a graph G was...
11/25/2021

Sketch-Guided Equality Saturation: Scaling Equality Saturation to Complex Optimizations in Languages with Bindings

Equality saturation is a technique for implementing rewrite-driven compi...
02/13/2018

Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

Deep learning models with convolutional and recurrent networks are now u...
11/23/2021

Caviar: An E-graph Based TRS for Automatic Code Optimization

Term Rewriting Systems (TRS) are used in compilers to simplify and prove...
06/07/2021

Learning Combinatorial Node Labeling Algorithms

We present a graph neural network to learn graph coloring heuristics usi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.