DeepAI AI Chat
Log In Sign Up

What Distributed Systems Say: A Study of Seven Spark Application Logs

08/18/2021
by   Sina Gholamian, et al.
0

Execution logs are a crucial medium as they record runtime information of software systems. Although extensive logs are helpful to provide valuable details to identify the root cause in postmortem analysis in case of a failure, this may also incur performance overhead and storage cost. Therefore, in this research, we present the result of our experimental study on seven Spark benchmarks to illustrate the impact of different logging verbosity levels on the execution time and storage cost of distributed software systems. We also evaluate the log effectiveness and the information gain values, and study the changes in performance and the generated logs for each benchmark with various types of distributed system failures. Our research draws insightful findings for developers and practitioners on how to set up and utilize their distributed systems to benefit from the execution logs.

READ FULL TEXT

page 7

page 9

11/08/2018

Tools and Benchmarks for Automated Log Parsing

Logs are imperative in the development and maintenance process of many s...
01/09/2023

Making Sense of Failure Logs in an Industrial DevOps Environment

Processing and reviewing nightly test execution failure logs for large i...
06/28/2021

Revelio: ML-Generated Debugging Queries for Distributed Systems

A major difficulty in debugging distributed systems lies in manually det...
01/08/2020

Comparing Constraints Mined From Execution Logs to Understand Software Evolution

Complex software systems evolve frequently, e.g., when introducing new f...
09/24/2019

Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

System logs record detailed runtime information of software systems and ...
12/22/2021

Log severity level classification: an approach for systems in production

Context: Logs are often the primary source of information for system dev...
06/12/2020

Hindsight Logging for Model Training

Due to the long time-lapse between the triggering and detection of a bug...