Machine Learning-based Selection of Graph Partitioning Strategy Using the Characteristics of Graph Data and Algorithm

by   YoungJoon Park, et al.

Analyzing large graph data is an essential part of many modern applications, such as social networks. Due to its large computational complexity, distributed processing is frequently employed. This requires graph data to be divided across nodes, and the choice of partitioning strategy has a great impact on the execution time of the task. Yet, there is no one-size-fits-all partitioning strategy that performs well on arbitrary graph data and algorithms. The performance of a strategy depends on the characteristics of the graph data and algorithms. Moreover, due to the complexity of graph data and algorithms, manually identifying the best partitioning strategy is also infeasible. In this work, we propose a machine learning-based approach to select the most appropriate partitioning strategy for a given graph and processing algorithm. Our approach enumerates viable partitioning strategies, predicts the execution time of the target algorithm for each, and selects the partitioning strategy with the fastest estimated execution time. Our machine learning model is trained on features extracted from graph data and algorithm pseudo-code. We also propose a method that augments real execution logs of graph tasks to create a large synthetic dataset. Evaluation results show that the strategies selected by our approach lead to 1.46X faster execution time on average compared with the mean execution time of the partitioning strategies and about 0.95X the performance compared to the best partitioning strategy.


Cut to Fit: Tailoring the Partitioning to the Computation

Social Graph Analytics applications are very often built using off-the-s...

Simulating Execution Time of Tensor Programs using Graph Neural Networks

Optimizing the execution time of tensor program, e.g., a convolution, in...

The TensorFlow Partitioning and Scheduling Problem: It's the Critical Path!

State-of-the-art data flow systems such as TensorFlow impose iterative c...

Accelerating Partial Evaluation in Distributed SPARQL Query Evaluation

Partial evaluation has recently been used for processing SPARQL queries ...

Partitioning SKA Dataflows for Optimal Graph Execution

Optimizing data-intensive workflow execution is essential to many modern...

Tuning symplectic integrators is easy and worthwhile

Many applications in computational physics that use numerical integrator...

On Optimizing Distributed Tucker Decomposition for Sparse Tensors

The Tucker decomposition generalizes the notion of Singular Value Decomp...

Please sign up or login with your details

Forgot password? Click here to reset