UniGPS: A Unified Programming Framework for Distributed Graph Processing

by   Zhaokang Wang, et al.

The industry and academia have proposed many distributed graph processing systems. However, the existing systems are not friendly enough for users like data analysts and algorithm engineers. On the one hand, the programing models and interfaces differ a lot in the existing systems, leading to high learning costs and program migration costs. On the other hand, these graph processing systems are tightly bound to the underlying distributed computing platforms, requiring users to be familiar with distributed computing. To improve the usability of distributed graph processing, we propose a unified distributed graph programming framework UniGPS. Firstly, we propose a unified cross-platform graph programming model VCProg for UniGPS. VCProg hides details of distributed computing from users. It is compatible with the popular graph programming models Pregel, GAS, and Push-Pull. VCProg programs can be executed by compatible distributed graph processing systems without modification, reducing the learning overheads of users. Secondly, UniGPS supports Python as the programming language. We propose an interprocess-communication-based execution environment isolation mechanism to enable Java/C++-based systems to call user-defined methods written in Python. The experimental results show that UniGPS enables users to process big graphs beyond the memory capacity of a single machine without sacrificing usability. UniGPS shows near-linear data scalability and machine scalability.


Quegel: A General-Purpose Query-Centric Framework for Querying Big Graphs

Pioneered by Google's Pregel, many distributed systems have been develop...

Extended Abstract: Productive Parallel Programming with Parsl

Parsl is a parallel programming library for Python that aims to make it ...

Graph3S: A Simple, Speedy and Scalable Distributed Graph Processing System

Graph is a ubiquitous structure in many domains. The rapidly increasing ...

Development details and computational benchmarking of DEPAM

In the big data era of observational oceanography, passive acoustics dat...

Kudu: An Efficient and Scalable Distributed Graph Pattern Mining Engine

This paper proposes Kudu, a general distributed execution engine with a ...

Got: Git, but for Objects

We look at one important category of distributed applications characteri...

To Push or To Pull: On Reducing Communication and Synchronization in Graph Computations

We reduce the cost of communication and synchronization in graph process...

Please sign up or login with your details

Forgot password? Click here to reset