Towards Scalable Subscription Aggregation and Real Time Event Matching in a Large-Scale Content-Based Network

11/17/2018
by   Ruisheng Shi, et al.
0

Although many scalable event matching algorithms have been proposed to achieve scalability for large-scale content-based networks, content-based publish/subscribe networks (especially for large-scale real time systems) still suffer performance deterioration when subscription scale increases. While subscription aggregation techniques can be useful to reduce the amount of subscription dissemination traffic and the subscription table size by exploiting the similarity among subscriptions, efficient subscription aggregation is not a trivial task to accomplish. Previous research works have proved that it is either a NP-Complete or a co-NP complete problem. In this paper, we propose DLS (Discrete Label Set), a novel subscription representation model, and design algorithms to achieve the mapping from traditional Boolean predicate model to the DLS model. Based on the DLS model, we propose a subscription aggregation algorithm with O(1) time complexity in most cases, and an event matching algorithm with O(1) time complexity. The significant performance improvement is at the cost of memory consumption and controllable false positive rate. Our theoretical analysis shows that these algorithms are inherently scalable and can achieve real time event matching in a large-scale content-based publish/subscribe network. We discuss the tradeoff between memory, false positive rate and partition granules of content space. Experimental results show that proposed algorithms achieve expected performance. With the increasing of computer memory capacity and the dropping of memory price, more and more large-scale real time applications can benefit from our proposed DLS model, such as stock quote distribution, earthquake monitoring, and severe weather alert.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/11/2021

Smart Name Lookup for NDN Forwarding Plane via Neural Networks

Name lookup is a key technology for the forwarding plane of content rout...
06/14/2020

A Neural Network Approach for Online Nonlinear Neyman-Pearson Classification

We propose a novel Neyman-Pearson (NP) classifier that is both online an...
01/15/2018

tau-FPL: Tolerance-Constrained Learning in Linear Time

Learning a classifier with control on the false-positive rate plays a cr...
07/22/2021

Pre-Clustering Point Clouds of Crop Fields Using Scalable Methods

In order to apply the recent successes of automated plant phenotyping an...
08/25/2020

Efficient Hierarchical Clustering for Classification and Anomaly Detection

We address the problem of large scale real-time classification of conten...
11/10/2017

Real-time Stream-based Monitoring

We introduce RTLola, a new stream-based specification language for the d...
06/29/2018

Topology classification with deep learning to improve real-time event selection at the LHC

We show how event topology classification based on deep learning could b...