DeepAI AI Chat
Log In Sign Up

What Distributed Systems Say: A Study of Seven Spark Application Logs

by   Sina Gholamian, et al.

Execution logs are a crucial medium as they record runtime information of software systems. Although extensive logs are helpful to provide valuable details to identify the root cause in postmortem analysis in case of a failure, this may also incur performance overhead and storage cost. Therefore, in this research, we present the result of our experimental study on seven Spark benchmarks to illustrate the impact of different logging verbosity levels on the execution time and storage cost of distributed software systems. We also evaluate the log effectiveness and the information gain values, and study the changes in performance and the generated logs for each benchmark with various types of distributed system failures. Our research draws insightful findings for developers and practitioners on how to set up and utilize their distributed systems to benefit from the execution logs.


page 7

page 9


Tools and Benchmarks for Automated Log Parsing

Logs are imperative in the development and maintenance process of many s...

Making Sense of Failure Logs in an Industrial DevOps Environment

Processing and reviewing nightly test execution failure logs for large i...

Revelio: ML-Generated Debugging Queries for Distributed Systems

A major difficulty in debugging distributed systems lies in manually det...

Comparing Constraints Mined From Execution Logs to Understand Software Evolution

Complex software systems evolve frequently, e.g., when introducing new f...

Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

System logs record detailed runtime information of software systems and ...

Log severity level classification: an approach for systems in production

Context: Logs are often the primary source of information for system dev...

Hindsight Logging for Model Training

Due to the long time-lapse between the triggering and detection of a bug...