DeepAI AI Chat
Log In Sign Up

Testing SOAR Tools in Use

by   Robert A. Bridges, et al.

Modern security operation centers (SOCs) rely on operators and a tapestry of logging and alerting tools with large scale collection and query abilities. SOC investigations are tedious as they rely on manual efforts to query diverse data sources, overlay related logs, and correlate the data into information and then document results in a ticketing system. Security orchestration, automation, and response (SOAR) tools are a new technology that promise to collect, filter, and display needed data; automate common tasks that require SOC analysts' time; facilitate SOC collaboration; and, improve both efficiency and consistency of SOCs. SOAR tools have never been tested in practice to evaluate their effect and understand them in use. In this paper, we design and administer the first hands-on user study of SOAR tools, involving 24 participants and 6 commercial SOAR tools. Our contributions include the experimental design, itemizing six characteristics of SOAR tools and a methodology for testing them. We describe configuration of the test environment in a cyber range, including network, user, and threat emulation; a full SOC tool suite; and creation of artifacts allowing multiple representative investigation scenarios to permit testing. We present the first research results on SOAR tools. We found that SOAR configuration is critical, as it involves creative design for data display and automation. We found that SOAR tools increased efficiency and reduced context switching during investigations, although ticket accuracy and completeness (indicating investigation quality) decreased with SOAR use. Our findings indicated that user preferences are slightly negatively correlated with their performance with the tool; overautomation was a concern of senior analysts, and SOAR tools that balanced automation with assisting a user to make decisions were preferred.


page 6

page 16

page 18

page 19

page 20

page 24

page 36


Can Commercial Testing Automation Tools Work for IoT? A Case Study of Selenium and Node-Red

Background: Testing IoT software is challenging due to large scale, volu...

CDST: A Toolkit for Testing Cockpit Display Systems of Avionics

Avionics are highly critical systems that require extensive testing gove...

Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows

Efforts to make machine learning more widely accessible have led to a ra...

A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data

Security operation centers (SOCs) all over the world are tasked with rea...

Forming IDEAS Interactive Data Exploration & Analysis System

Modern cyber security operations collect an enormous amount of logging a...

Pandora: A Cyber Range Environment for the Safe Testing and Deployment of Autonomous Cyber Attack Tools

Cybersecurity tools are increasingly automated with artificial intellige...

How do information security workers use host data? A summary of interviews with security analysts

Modern security operations centers (SOCs) employ a variety of tools for ...