Robust Data Preprocessing for Machine-Learning-Based Disk Failure Prediction in Cloud Production Environments

12/20/2019
by   Shujie Han, et al.
0

To provide proactive fault tolerance for modern cloud data centers, extensive studies have proposed machine learning (ML) approaches to predict imminent disk failures for early remedy and evaluated their approaches directly on public datasets (e.g., Backblaze SMART logs). However, in real-world production environments, the data quality is imperfect (e.g., inaccurate labeling, missing data samples, and complex failure types), thereby degrading the prediction accuracy. We present RODMAN, a robust data preprocessing pipeline that refines data samples before feeding them into ML models. We start with a large-scale trace-driven study of over three million disks from Alibaba Cloud's data centers, and motivate the practical challenges in ML-based disk failure prediction. We then design RODMAN with three data preprocessing echniques, namely failure-type filtering, spline-based data filling, and automated pre-failure backtracking, that are applicable for general ML models. Evaluation on both the Alibaba and Backblaze datasets shows that RODMAN improves the prediction accuracy compared to without data preprocessing under various settings.

READ FULL TEXT
research
11/18/2019

Towards Quantification of Bias in Machine Learning for Healthcare: A Case Study of Renal Failure Prediction

As machine learning (ML) models, trained on real-world datasets, become ...
research
08/23/2022

A Review of Machine Learning-based Failure Management in Optical Networks

Failure management plays a significant role in optical networks. It ensu...
research
03/19/2021

Empirical Analysis of Machine Learning Configurations for Prediction of Multiple Organ Failure in Trauma Patients

Multiple organ failure (MOF) is a life-threatening condition. Due to its...
research
02/02/2023

MLOps with enhanced performance control and observability

The explosion of data and its ever increasing complexity in the last few...
research
04/24/2021

Highly Efficient Memory Failure Prediction using Mcelog-based Data Mining and Machine Learning

In the data center, unexpected downtime caused by memory failures can le...
research
10/05/2022

SECOE: Alleviating Sensors Failure in Machine Learning-Coupled IoT Systems

Machine learning (ML) applications continue to revolutionize many domain...
research
11/04/2019

A Spark ML driven preprocessing approach for deep learning based scholarly data applications

Big data has found applications in multiple domains. One of the largest ...

Please sign up or login with your details

Forgot password? Click here to reset