Distributed Evaluation of Subgraph Queries Using Worstcase Optimal LowMemory Dataflows

02/11/2018
by   Khaled Ammar, et al.
0

We study the problem of finding and monitoring fixed-size subgraphs in a continually changing large-scale graph. We present the first approach that (i) performs worst-case optimal computation and communication, (ii) maintains a total memory footprint linear in the number of input edges, and (iii) scales down per-worker computation, communication, and memory requirements linearly as the number of workers increases, even on adversarially skewed inputs. Our approach is based on worst-case optimal join algorithms, recast as a data-parallel dataflow computation. We describe the general algorithm and modifications that make it robust to skewed data, prove theoretical bounds on its resource requirements in the massively parallel computing model, and implement and evaluate it on graphs containing as many as 64 billion edges. The underlying algorithm and ideas generalize from finding and monitoring subgraphs to the more general problem of computing and maintaining relational equi-joins over dynamic relations.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/27/2021

Efficient Parallel Graph Trimming by Arc-Consistency

Given a large data graph, trimming techniques can reduce the search spac...
05/04/2021

Finding Triangles or Independent Sets

(I) We revisit the algorithmic problem of finding all triangles in a gra...
12/29/2019

Worst-Case Optimal Radix Triejoin

Relatively recently, the field of join processing has been swayed by the...
08/05/2019

Optimal Joins using Compact Data Structures

Worst-case optimal join algorithms have gained a lot of attention in the...
02/01/2018

Distributed Computing with Heterogeneous Communication Constraints: The Worst-Case Computation Load and Proof by Contradiction

We consider a distributed computing framework where the distributed node...
07/31/2018

Composable Core-sets for Determinant Maximization Problems via Spectral Spanners

We study a spectral generalization of classical combinatorial graph span...
08/20/2021

Federated Distributionally Robust Optimization for Phase Configuration of RISs

In this article, we study the problem of robust reconfigurable intellige...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.