On the Scalability of Big Data Cyber Security Analytics Systems
share
Big Data Cyber Security Analytics (BDCA) systems use big data technologies (e.g., Apache Spark) to collect, store, and analyze a large volume of security event data for detecting cyber-attacks. The volume of digital data in general and security event data in specific is increasing exponentially. The velocity with which the security event data is generated and fed into a BDCA system is unpredictable. Therefore, a BDCA system should be highly scalable to deal with the unpredictable increase/decrease in the velocity of security event data. However, there has been little effort to investigate the scalability of BDCA systems to identify and exploit the sources of scalability improvement. In this paper, we first investigate the scalability of a Spark-based BDCA system with default Spark settings. we then identify Spark configuration parameters (e.g., execution memory) that can significantly impact the scalability of a BDCA system. Based on the identified parameters, we finally propose a parameter-driven adaptation approach, SCALER, for optimizing a system's scalability. We have conducted a set of experiments by implementing a Spark-based BDCA system on a large-scale OpenStack cluster. We ran our experiments with four security datasets. We have found that (i) a BDCA system with default Spark configuration parameters deviates from ideal scalability by 59.5 impact scalability (iii) SCALER improves the BDCA system's scalability by 20.8 compared to the scalability with default Spark parameter setting. The findings of our study highlight the importance of exploring the parameter space of the underlying big data framework (e.g., Apache Spark) for scalable cyber security analytics.
READ FULL TEXT
