DeepAI AI Chat
Log In Sign Up

Streaming Graph Challenge: Stochastic Block Partition

by   Edward Kao, et al.

An important objective for analyzing real-world graphs is to achieve scalable performance on large, streaming graphs. A challenging and relevant example is the graph partition problem. As a combinatorial problem, graph partition is NP-hard, but existing relaxation methods provide reasonable approximate solutions that can be scaled for large graphs. Competitive benchmarks and challenges have proven to be an effective means to advance state-of-the-art performance and foster community collaboration. This paper describes a graph partition challenge with a baseline partition algorithm of sub-quadratic complexity. The algorithm employs rigorous Bayesian inferential methods based on a statistical model that captures characteristics of the real-world graphs. This strong foundation enables the algorithm to address limitations of well-known graph partition approaches such as modularity maximization. This paper describes various aspects of the challenge including: (1) the data sets and streaming graph generator, (2) the baseline partition algorithm with pseudocode, (3) an argument for the correctness of parallelizing the Bayesian inference, (4) different parallel computation strategies such as node-based parallelism and matrix-based parallelism, (5) evaluation metrics for partition correctness and computational requirements, (6) preliminary timing of a Python-based demonstration code and the open source C++ code, and (7) considerations for partitioning the graph in streaming fashion. Data sets and source code for the algorithm as well as metrics, with detailed documentation are available at


Hybrid Edge Partitioner: Partitioning Large Power-Law Graphs under Memory Constraints

Distributed systems that manage and process graph-structured data intern...

Preconditioned Spectral Clustering for Stochastic Block Partition Streaming Graph Challenge

Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) is demo...

Analysis of large sparse graphs using regular decomposition of graph distance matrices

Statistical analysis of large and sparse graphs is a challenging problem...

Buffered Streaming Graph Partitioning

Partitioning graphs into blocks of roughly equal size is a widely used t...

Evolutionary Acyclic Graph Partitioning

Directed graphs are widely used to model data flow and execution depende...

Flow-Partitionable Signed Graphs

The NP-hard problem of correlation clustering is to partition a signed g...

Computational graphs for matrix functions

Many numerical methods for evaluating matrix functions can be naturally ...