DeepAI AI Chat
Log In Sign Up

Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings

by   Hanzhang Wang, et al.

For large-scale distributed systems, it's crucial to efficiently diagnose the root causes of incidents to maintain high system availability. The recent development of microservice architecture brings three major challenges (i.e., operation, system scale, and monitoring complexities) to root cause analysis (RCA) in industrial settings. To tackle these challenges, in this paper, we present Groot, an event-graph-based approach for RCA. Groot constructs a real-time causality graph based on events that summarize various types of metrics, logs, and activities in the system under analysis. Moreover, to incorporate domain knowledge from site reliability engineering (SRE) engineers, Groot can be customized with user-defined events and domain-specific rules. Currently, Groot supports RCA among 5,000 real production services and is actively used by the SRE teamin a global e-commerce system serving more than 185 million active buyers per year. Over 15 months, we collect a data setcontaining labeled root causes of 952 real production incidents for evaluation. The evaluation results show that Groot is able to achieve 95 accuracy and 78 adopting RCA in industrial settings, we conduct survey to show that users of Grootfindit helpful and easy to use. We also share the lessons learnedfrom deploying and adopting Grootto solve RCA problems inproduction environments.


page 1

page 9


MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Availability issues of industrial microservice systems (e.g., drop of su...

An Event Based Approach To Situational Representation

Many application domains require representing interrelated real-world ac...

GraphLED: A graph-based approach to process and visualise linked engineering documents

The architecture, engineering and construction (AEC) sector extensively ...

Fast Dimensional Analysis for Root Cause Investigation in a Large-Scale Service Environment

Root cause analysis in a large-scale production environment is challengi...

Making Sense of Failure Logs in an Industrial DevOps Environment

Processing and reviewing nightly test execution failure logs for large i...

Live Forensics for Distributed Storage Systems

We present Kaleidoscope an innovative system that supports live forensic...

Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models

Incident management for cloud services is a complex process involving se...