Window-based Streaming Graph Partitioning Algorithm

02/05/2019
by   Md Anwarul kaium Patwary, et al.
0

In the recent years, the scale of graph datasets has increased to such a degree that a single machine is not capable of efficiently processing large graphs. Thereby, efficient graph partitioning is necessary for those large graph applications. Traditional graph partitioning generally loads the whole graph data into the memory before performing partitioning; this is not only a time consuming task but it also creates memory bottlenecks. These issues of memory limitation and enormous time complexity can be resolved using stream-based graph partitioning. A streaming graph partitioning algorithm reads vertices once and assigns that vertex to a partition accordingly. This is also called an one-pass algorithm. This paper proposes an efficient window-based streaming graph partitioning algorithm called WStream. The WStream algorithm is an edge-cut partitioning algorithm, which distributes a vertex among the partitions. Our results suggest that the WStream algorithm is able to partition large graph data efficiently while keeping the load balanced across different partitions, and communication to a minimum. Evaluation results with real workloads also prove the effectiveness of our proposed algorithm, and it achieves a significant reduction in load imbalance and edge-cut with different ranges of dataset.

READ FULL TEXT
research
06/18/2018

VEBO: A Vertex- and Edge-Balanced Ordering Heuristic to Load Balance Parallel Graph Processing

Graph partitioning drives graph processing in distributed, disk-based an...
research
10/29/2021

SDP: Scalable Real-time Dynamic Graph Partitioner

Time-evolving large graph has received attention due to their participat...
research
01/03/2022

Clustering-based Partitioning for Large Web Graphs

Graph partitioning plays a vital role in distributedlarge-scale web grap...
research
07/15/2019

Partitioning Graphs for the Cloud using Reinforcement Learning

In this paper, we propose Revolver, a parallel graph partitioning algori...
research
04/12/2018

Seed-Point Based Geometric Partitioning of Nuclei Clumps

When applying automatic analysis of fluorescence or histopathological im...
research
03/02/2019

GAP: Generalizable Approximate Graph Partitioning Framework

Graph partitioning is the problem of dividing the nodes of a graph into ...
research
07/06/2020

Prioritized Restreaming Algorithms for Balanced Graph Partitioning

Balanced graph partitioning is a critical step for many large-scale dist...

Please sign up or login with your details

Forgot password? Click here to reset