Modeling Updates of Scholarly Webpages Using Archived Data

by   Yasith Jayawardana, et al.

The vastness of the web imposes a prohibitive cost on building large-scale search engines with limited resources. Crawl frontiers thus need to be optimized to improve the coverage and freshness of crawled content. In this paper, we propose an approach for modeling the dynamics of change in the web using archived copies of webpages. To evaluate its utility, we conduct a preliminary study on the scholarly web using 19,977 seed URLs of authors' homepages obtained from their Google Scholar profiles. We first obtain archived copies of these webpages from the Internet Archive (IA), and estimate when their actual updates occurred. Next, we apply maximum likelihood to estimate their mean update frequency (λ) values. Our evaluation shows that λ values derived from a short history of archived data provide a good estimate for the true update frequency in the short-term, and that our method provides better estimations of updates at a fraction of resources compared to the baseline models. Based on this, we demonstrate the utility of archived data to optimize the crawling strategy of web crawlers, and uncover important challenges that inspire future research directions.


Protecting User Privacy: An Approach for Untraceable Web Browsing History and Unambiguous User Profiles

The overturning of the Internet Privacy Rules by the Federal Communicati...

How to Search the Internet Archive Without Indexing It

Significant parts of cultural heritage are produced on the web during th...

Readability and Understandability of Snippets Recommended by General-purpose Web Search Engines: a Comparative Study

Developers often search for reusable code snippets on general-purpose we...

Exploring Web Search Engines to Find Architectural Knowledge

Software engineers need relevant and up-to-date architectural knowledge ...

Secure OTA Software Updates in Connected Vehicles: A survey

This survey highlights and discusses remote OTA software updates in the ...

Development and evaluation of an open-source, machine learning-based average annual daily traffic estimation software

Traditionally, Departments of Transportation (DOTs) use the factor-based...

Private Federated Submodel Learning with Sparsification

We investigate the problem of private read update write (PRUW) in federa...