Web Robot Detection in Academic Publishing

by   Athanasios Lagopoulos, et al.

Recent industry reports assure the rise of web robots which comprise more than half of the total web traffic. They not only threaten the security, privacy and efficiency of the web but they also distort analytics and metrics, doubting the veracity of the information being promoted. In the academic publishing domain, this can cause articles to be faulty presented as prominent and influential. In this paper, we present our approach on detecting web robots in academic publishing websites. We use different supervised learning algorithms with a variety of characteristics deriving from both the log files of the server and the content served by the website. Our approach relies on the assumption that human users will be interested in specific domains or articles, while web robots crawl a web library incoherently. We experiment with features adopted in previous studies with the addition of novel semantic characteristics which derive after performing a semantic analysis using the Latent Dirichlet Allocation (LDA) algorithm. Our real-world case study shows promising results, pinpointing the significance of semantic features in the web robot detection problem.


page 1

page 2

page 3

page 4


Dot-Science Top Level Domain: academic websites or dumpsites?

Dot-science was launched in 2015 as a new academic top-level domain (TLD...

Realistic Traffic Generation for Web Robots

Critical to evaluating the capacity, scalability, and availability of we...

Two Novel Server-Side Attacks against Log File in Shared Web Hosting Servers

Shared Web Hosting service enables hosting multitude of websites on a si...

Calculating Semantic Similarity between Academic Articles using Topic Event and Ontology

Determining semantic similarity between academic documents is crucial to...

Approximating percentage of academic traffic in the World Wide Web and rankings of countries based on academic traffic

The paper introduces a novel mechanism for approximating traffic of the ...

Similarity-based web element localization for robust test automation

Non-robust (fragile) test execution is a commonly reported challenge in ...