An Assessment of the Usability of Machine Learning Based Tools for the Security Operations Center

by   Sean Oesch, et al.

Gartner, a large research and advisory company, anticipates that by 2024 80 of security operation centers (SOCs) will use machine learning (ML) based solutions to enhance their operations. In light of such widespread adoption, it is vital for the research community to identify and address usability concerns. This work presents the results of the first in situ usability assessment of ML-based tools. With the support of the US Navy, we leveraged the national cyber range, a large, air-gapped cyber testbed equipped with state-of-the-art network and user emulation capabilities, to study six US Naval SOC analysts' usage of two tools. Our analysis identified several serious usability issues, including multiple violations of established usability heuristics form user interface design. We also discovered that analysts lacked a clear mental model of how these tools generate scores, resulting in mistrust and/or misuse of the tools themselves. Surprisingly, we found no correlation between analysts' level of education or years of experience and their performance with either tool, suggesting that other factors such as prior background knowledge or personality play a significant role in ML-based tool usage. Our findings demonstrate that ML-based security tool vendors must put a renewed focus on working with analysts, both experienced and inexperienced, to ensure that their systems are usable and useful in real-world security operations settings.


page 1

page 5


Usable Security for ML Systems in Mental Health: A Framework

While the applications and demands of Machine learning (ML) systems in m...

A Neophyte With AutoML: Evaluating the Promises of Automatic Machine Learning Tools

This paper discusses modern Auto Machine Learning (AutoML) tools from th...

Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making

Machine learning (ML) is being applied to a diverse and ever-growing set...

Machine Learning Operations: A Survey on MLOps Tool Support

Machine Learning (ML) has become a fast-growing, trending approach in so...

A methodology to Evaluate the Usability of Security APIs

Increasing number of cyber-attacks demotivate people to use Information ...

How do information security workers use host data? A summary of interviews with security analysts

Modern security operations centers (SOCs) employ a variety of tools for ...

A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data

Security operation centers (SOCs) all over the world are tasked with rea...

Please sign up or login with your details

Forgot password? Click here to reset