A Hierarchical Approach to Conditional Random Fields for System Anomaly Detection

by   Srishti Mishra, et al.

Anomaly detection to recognize unusual events in large scale systems in a time sensitive manner is critical in many industries, eg. bank fraud, enterprise systems, medical alerts, etc. Large-scale systems often grow in size and complexity over time, and anomaly detection algorithms need to adapt to changing structures. A hierarchical approach takes advantage of the implicit relationships in complex systems and localized context. The features in complex systems may vary drastically in data distribution, capturing different aspects from multiple data sources, and when put together provide a more complete view of the system. In this paper, two datasets are considered, the 1st comprising of system metrics from machines running on a cloud service, and the 2nd of application metrics from a large-scale distributed software system with inherent hierarchies and interconnections amongst its system nodes. Comparing algorithms, across the changepoint based PELT algorithm, cognitive learning-based Hierarchical Temporal Memory algorithms, Support Vector Machines and Conditional Random Fields provides a basis for proposing a Hierarchical Global-Local Conditional Random Field approach to accurately capture anomalies in complex systems across various features. Hierarchical algorithms can learn both the intricacies of specific features, and utilize these in a global abstracted representation to detect anomalous patterns robustly across multi-source feature data and distributed systems. A graphical network analysis on complex systems can further fine-tune datasets to mine relationships based on available features, which can benefit hierarchical models. Furthermore, hierarchical solutions can adapt well to changes at a localized level, learning on new data and changing environments when parts of a system are over-hauled, and translate these learnings to a global view of the system over time.


Federated Multi-Discriminator BiWGAN-GP based Collaborative Anomaly Detection for Virtualized Network Slicing

Virtualized network slicing allows a multitude of logical networks to be...

Performance Issue Identification in Cloud Systems with Relational-Temporal Anomaly Detection

Performance issues permeate large-scale cloud service systems, which can...

Practical data monitoring in the internet-services domain

Large-scale monitoring, anomaly detection, and root cause analysis of me...

Practical Anomaly Detection over Multivariate Monitoring Metrics for Online Services

As modern software systems continue to grow in terms of complexity and v...

Heterogeneous Anomaly Detection for Software Systems via Attentive Multi-modal Learning

Prompt and accurate detection of system anomalies is essential to ensure...

Probabilistic Graphs for Sensor Data-driven Modelling of Power Systems at Scale

The growing complexity of the power grid, driven by increasing share of ...

Distributed Online Anomaly Detection for Virtualized Network Slicing Environment

As the network slicing is one of the critical enablers in communication ...

Please sign up or login with your details

Forgot password? Click here to reset