Fidelity of Statistical Reporting in 10 Years of Cyber Security User Studies

04/14/2020
by   Thomas Groß, et al.
0

Studies in socio-technical aspects of security often rely on user studies and statistical inferences on investigated relations to make their case. They, thereby, enable practitioners and scientists alike to judge on the validity and reliability of the research undertaken. To ascertain this capacity, we investigated the reporting fidelity of security user studies. Based on a systematic literature review of 114 user studies in cyber security from selected venues in the 10 years 2006–2016, we evaluated fidelity of the reporting of 1775 statistical inferences using the R package statcheck. We conducted a systematic classification of incomplete reporting, reporting inconsistencies and decision errors, leading to multinomial logistic regression (MLR) on the impact of publication venue/year as well as a comparison to a compatible field of psychology. We found that half the cyber security user studies considered reported incomplete results, in stark difference to comparable results in a field of psychology. Our MLR on analysis outcomes yielded a slight increase of likelihood of incomplete tests over time, while SOUPS yielded a few percent greater likelihood to report statistics correctly than other venues. In this study, we offer the first fully quantitative analysis of the state-of-play of socio-technical studies in security. While we highlight the impact and prevalence of incomplete reporting, we also offer fine-grained diagnostics and recommendations on how to respond to the situation.

READ FULL TEXT

page 13

page 15

research
10/05/2020

Statistical Reliability of 10 Years of Cyber Security User Studies (Extended Version)

Background. In recent years, cyber security security user studies have b...
research
01/22/2022

What and How Are We Reporting in HRI? A Review and Recommendations for Reporting Recruitment, Compensation, and Gender

Study reproducibility and generalizability of results to broadly inclusi...
research
09/04/2023

A Systematic Review on Reproducibility in Child-Robot Interaction

Research reproducibility - i.e., rerunning analyses on original data to ...
research
01/20/2022

Correcting for Reporting Delays in Cyber Incidents

With an ever evolving cyber domain, delays in reporting incidents are a ...
research
09/19/2020

A framework for effective corporate communication after cyber security incidents

A major cyber security incident can represent a cyber crisis for an orga...
research
07/28/2019

A Systematic Review of Unsupervised Learning Techniques for Software Defect Prediction

Background: Unsupervised machine learners have been increasingly applied...
research
06/07/2018

Developing Cyber Peacekeeping: Observation, Monitoring and Reporting

Cyber peacekeeping is an emerging and multi-disciplinary field of resear...

Please sign up or login with your details

Forgot password? Click here to reset