Graphsurge: Graph Analytics on View Collections Using Differential Computation
share
This paper presents the design and implementation of a new open-source view-based graph analytics system called Graphsurge. Graphsurge is designed to support applications that analyze multiple snapshots or views of a large-scale graph. Users program Graphsurge through a declarative graph view definition language to create views over input graphs and a Differential Dataflow-based programming API to write analytics computations. A key feature of GVDL is the ability to organize views into view collections, which allows Graphsurge to share computation across views by performing computations differentially. We then introduce two optimization problems that naturally arises in our setting. First is the view ordering problem to determine the order of views that leads to minimum differences across consecutive views. We prove this problem is NP-hard and show a constant-factor approximation algorithm drawn from literature. Second is the collection splitting problem to decide on which views to run computations differentially vs from scratch, for which we present an adaptive solution that makes decisions at runtime. Graphsurge is implemented on top of the Timely and Differential Dataflow systems. We present extensive experiments to demonstrate the benefits of running computations differentially for view collections and our view ordering and collection splitting optimizations.
READ FULL TEXT