Analysis of Statistical Hypothesis based Learning Mechanism for Faster Crawling

08/10/2012
by   Sudarshan Nandy, et al.
0

The growth of world-wide-web (WWW) spreads its wings from an intangible quantities of web-pages to a gigantic hub of web information which gradually increases the complexity of crawling process in a search engine. A search engine handles a lot of queries from various parts of this world, and the answers of it solely depend on the knowledge that it gathers by means of crawling. The information sharing becomes a most common habit of the society, and it is done by means of publishing structured, semi-structured and unstructured resources on the web. This social practice leads to an exponential growth of web-resource, and hence it became essential to crawl for continuous updating of web-knowledge and modification of several existing resources in any situation. In this paper one statistical hypothesis based learning mechanism is incorporated for learning the behavior of crawling speed in different environment of network, and for intelligently control of the speed of crawler. The scaling technique is used to compare the performance proposed method with the standard crawler. The high speed performance is observed after scaling, and the retrieval of relevant web-resource in such a high speed is analyzed.

READ FULL TEXT
research
08/09/2023

web crawler strategies for web pages under robot.txt restriction

In the present time, all know about World Wide Web and work over the Int...
research
06/21/2023

Comparative analysis of various web crawler algorithms

This presentation focuses on the importance of web crawling and page ran...
research
03/10/2022

Evaluating Elements of Web-based Data Enrichment for Pseudo-Relevance Feedback Retrieval

In this work, we analyze a pseudo-relevance retrieval method based on th...
research
10/18/2018

Creating a Web Analysis and Visualization Environment

Due to the rapid growth of the World Wide Web, resource discovery become...
research
09/22/2017

Annotation based automatic action processing

With a strong motivational background in search engine optimization the ...
research
08/01/2019

A Hessenberg-type Algorithm for Computing PageRank Problems

PageRank is a greatly essential ranking algorithm in web information ret...
research
08/20/2018

FedMark: A Marketplace for Federated Data on the Web

The Web of Data (WoD) has experienced a phenomenal growth in the past. T...

Please sign up or login with your details

Forgot password? Click here to reset