LogLAB: Attention-Based Labeling of Log Data Anomalies via Weak Supervision

11/02/2021
by   Thorsten Wittkopp, et al.
0

With increasing scale and complexity of cloud operations, automated detection of anomalies in monitoring data such as logs will be an essential part of managing future IT infrastructures. However, many methods based on artificial intelligence, such as supervised deep learning models, require large amounts of labeled training data to perform well. In practice, this data is rarely available because labeling log data is expensive, time-consuming, and requires a deep understanding of the underlying system. We present LogLAB, a novel modeling approach for automated labeling of log messages without requiring manual work by experts. Our method relies on estimated failure time windows provided by monitoring systems to produce precise labeled datasets in retrospect. It is based on the attention mechanism and uses a custom objective function for weak supervision deep learning techniques that accounts for imbalanced data. Our evaluation shows that LogLAB consistently outperforms nine benchmark approaches across three different datasets and maintains an F1-score of more than 0.98 even at large failure time windows.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/25/2023

PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning

Due to the complexity of modern IT services, failures can be manifold, o...
research
02/09/2022

Log-based Anomaly Detection with Deep Learning: How Far Are We?

Software-intensive systems produce logs for troubleshooting purposes. Re...
research
06/09/2023

Automated Labeling of German Chest X-Ray Radiology Reports using Deep Learning

Radiologists are in short supply globally, and deep learning models offe...
research
12/09/2019

A Weak Supervision Approach to Detecting Visual Anomalies for Automated Testing of Graphics Units

We present a deep learning system for testing graphics units by detectin...
research
10/04/2019

Active Learning with Weak Supervision for Cost-Effective Panicle Detection in Cereal Crops

Panicle density of cereal crops such as wheat and sorghum is one of the ...
research
10/11/2022

Digitization of Raster Logs: A Deep Learning Approach

Raster well-log images are digital representations of well-logs data gen...

Please sign up or login with your details

Forgot password? Click here to reset