Streaming Hypergraph Partitioning Algorithms on Limited Memory Environments

03/09/2021
by   Fatih Taşyaran, et al.
0

Many well-known, real-world problems involve dynamic data which describe the relationship among the entities. Hypergraphs are powerful combinatorial structures that are frequently used to model such data. For many of today's data-centric applications, this data is streaming; new items arrive continuously, and the data grows with time. With paradigms such as Internet of Things and Edge Computing, such applications become more natural and more practical. In this work, we assume a streaming model where the data is modeled as a hypergraph, which is generated at the edge. This data then partitioned and sent to remote nodes via an algorithm running on a memory-restricted device such as a single board computer. Such a partitioning is usually performed by taking a connectivity metric into account to minimize the communication cost of later analyses that will be performed in a distributed fashion. Although there are many offline tools that can partition static hypergraphs excellently, algorithms for the streaming settings are rare. We analyze a well-known algorithm from the literature and significantly improve its running time by altering its inner data structure. For instance, on a medium-scale hypergraph, the new algorithm reduces the runtime from 17800 seconds to 10 seconds. We then propose sketch- and hash-based algorithms, as well as ones that can leverage extra memory to store a small portion of the data to enable the refinement of partitioning when possible. We experimentally analyze the performance of these algorithms and report their run times, connectivity metric scores, and memory uses on a high-end server and four different single-board computer architectures.

READ FULL TEXT

Authors

page 1

page 7

03/23/2021

Hybrid Edge Partitioner: Partitioning Large Power-Law Graphs under Memory Constraints

Distributed systems that manage and process graph-structured data intern...
02/10/2018

Network Flow-Based Refinement for Multilevel Hypergraph Partitioning

We present a refinement framework for multilevel hypergraph partitioning...
02/06/2020

Multilevel Acyclic Hypergraph Partitioning

A directed acyclic hypergraph is a generalized concept of a directed acy...
01/24/2019

Dolha - an Efficient and Exact Data Structure for Streaming Graphs

A streaming graph is a graph formed by a sequence of incoming edges with...
02/18/2021

Buffered Streaming Graph Partitioning

Partitioning graphs into blocks of roughly equal size is a widely used t...
07/03/2019

Evaluation of a Flow-Based Hypergraph Bipartitioning Algorithm

In this paper, we propose HyperFlowCutter, an algorithm for balanced hyp...
05/20/2022

Swim: A Runtime for Distributed Event-Driven Applications

Swim extends the actor model to support applications composed of linked ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.