Failure Identification from Unstable Log Data using Deep Learning

by   Jasmin Bogatinovski, et al.

The reliability of cloud platforms is of significant relevance because society increasingly relies on complex software systems running on the cloud. To improve it, cloud providers are automating various maintenance tasks, with failure identification frequently being considered. The precondition for automation is the availability of observability tools, with system logs commonly being used. The focus of this paper is log-based failure identification. This problem is challenging because of the instability of the log data and the incompleteness of the explicit logging failure coverage within the code. To address the two challenges, we present CLog as a method for failure identification. The key idea presented herein based is on our observation that by representing the log data as sequences of subprocesses instead of sequences of log events, the effect of the unstable log data is reduced. CLog introduces a novel subprocess extraction method that uses context-aware neural network and clustering methods to extract meaningful subprocesses. The direct modeling of log event contexts allows the identification of failures with respect to the abrupt context changes, addressing the challenge of insufficient logging failure coverage. Our experimental results demonstrate that the learned subprocesses representations reduce the instability in the input, allowing CLog to outperform the baselines on the failure identification subproblems - 1) failure detection by 9-24 score and 2) failure type identification by 7 Further analysis shows the existent negative correlation between the instability in the input event sequences and the detection performance in a model-agnostic manner.


A hybrid feature learning approach based on convolutional kernels for ATM fault prediction using event-log data

Predictive Maintenance (PdM) methods aim to facilitate the scheduling of...

Towards Runtime Verification via Event Stream Processing in Cloud Computing Infrastructures

Software bugs in cloud management systems often cause erratic behavior, ...

vue4logs – Automatic Structuring of Heterogeneous Computer System Logs

Computer system log data is commonly used in system monitoring, performa...

Run-time Failure Detection via Non-intrusive Event Analysis in a Large-Scale Cloud Computing Platform

Cloud computing systems fail in complex and unforeseen ways due to unexp...

Failure Analysis of Big Cloud Service Providers Prior to and During Covid-19 Period

Cloud services are important for societal function such as healthcare, c...

Survival Analysis on Structured Data using Deep Reinforcement Learning

Survival analysis is playing a major role in manufacturing sector by ana...

The Derivation of Failure Event Correlation Based on Shadowing Cross-Correlation

In this document we derive the mapping between the failure event correla...

Please sign up or login with your details

Forgot password? Click here to reset