Log In Sign Up

Fast Botnet Detection From Streaming Logs Using Online Lanczos Method

by   Zheng Chen, et al.

Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, real-time protection capability, ease of use and better security of sensitive data. Our algorithm is inspired by a Principal Component Analysis (PCA) to capture correlation in data, and we are first to recognize and adapt Lanczos method to improve the time complexity of PCA-based botnet detection from cubic to sub-cubic, which enables us to more accurately and sensitively detect botnets with sliding time windows rather than fixed time windows. We contribute a generalized online correlation matrix update formula, and a new termination condition for Lanczos iteration for our purpose based on error bound and non-decreasing eigenvalues of symmetric matrices. On our dataset of an ecommerce website logs, experiments show the time cost of Lanczos method with different time windows are consistently only 20


page 1

page 2

page 3

page 4


Eigenvectors from Eigenvalues Sparse Principal Component Analysis (EESPCA)

We present a novel technique for sparse principal component analysis. Th...

Near-Optimal Stochastic Approximation for Online Principal Component Estimation

Principal component analysis (PCA) has been a prominent tool for high-di...

Online Principal Component Analysis in High Dimension: Which Algorithm to Choose?

In the current context of data explosion, online techniques that do not ...

Railgun: streaming windows for mission critical systems

Some mission critical systems, such as fraud detection, require accurate...

Railgun: managing large streaming windows under MAD requirements

Some mission critical systems, e.g., fraud detection, require accurate, ...

An Online Algorithm for Nonparametric Correlations

Nonparametric correlations such as Spearman's rank correlation and Kenda...