Using Graph Neural Networks to model the performance of Deep Neural Networks

by   Shikhar Singh, et al.

With the unprecedented proliferation of machine learning software, there is an ever-increasing need to generate efficient code for such applications. State-of-the-art deep-learning compilers like TVM and Halide incorporate a learning-based performance model to search the space of valid implementations of a given deep learning algorithm. For a given application, the model generates a performance metric such as the run time without executing the application on hardware. Such models speed up the compilation process by obviating the need to benchmark an enormous number of candidate implementations, referred to as schedules, on hardware. Existing performance models employ feed-forward networks, recurrent networks, or decision tree ensembles to estimate the performance of different implementations of a neural network. Graphs present a natural and intuitive way to model deep-learning networks where each node represents a computational stage or operation. Incorporating the inherent graph structure of these workloads in the performance model can enable a better representation and learning of inter-stage interactions. The accuracy of a performance model has direct implications on the efficiency of the search strategy, making it a crucial component of this class of deep-learning compilers. In this work, we develop a novel performance model that adopts a graph representation. In our model, each stage of computation represents a node characterized by features that capture the operations performed by the stage. The interaction between nodes is achieved using graph convolutions. Experimental evaluation shows a 7:75x and 12x reduction in prediction error compared to the Halide and TVM models, respectively.


Inductive Graph Representation Learning with Recurrent Graph Neural Networks

In this paper, we study the problem of node representation learning with...

Representation Learning on Graphs with Jumping Knowledge Networks

Recent deep learning approaches for representation learning on graphs fo...

Generalizable Machine Learning in Neuroscience using Graph Neural Networks

Although a number of studies have explored deep learning in neuroscience...

DEMO-Net: Degree-specific Graph Neural Networks for Node and Graph Classification

Graph data widely exist in many high-impact applications. Inspired by th...

Sparse Coding Driven Deep Decision Tree Ensembles for Nuclear Segmentation in Digital Pathology Images

In this paper, we propose an easily trained yet powerful representation ...

Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

The choice of convolutional routines (primitives) to implement neural ne...

Empirical Performance Analysis of Conventional Deep Learning Models for Recognition of Objects in 2-D Images

Artificial Neural Networks, an essential part of Deep Learning, are deri...