DeepAI AI Chat
Log In Sign Up

The Impact of Main Content Extraction on Near-Duplicate Detection

by   Maik Fröbe, et al.

Commercial web search engines employ near-duplicate detection to ensure that users see each relevant result only once, albeit the underlying web crawls typically include (near-)duplicates of many web pages. We revisit the risks and potential of near-duplicates with an information retrieval focus, motivating that current efforts toward an open and independent European web search infrastructure should maintain metadata on duplicate and near-duplicate documents in its index. Near-duplicate detection implemented in an open web search infrastructure should provide a suitable similarity threshold, a difficult choice since identical pages may substantially differ in parts of a page that are irrelevant to searchers (templates, advertisements, etc.). We study this problem by comparing the similarity of pages for five (main) content extraction methods in two studies on the ClueWeb crawls. We find that the full content of pages serves precision-oriented near-duplicate-detection, while main content extraction is more recall-oriented.


page 1

page 2

page 3

page 4


A Fast Template-based Approach to Automatically Identify Primary Text Content of a Web Page

Search engines have become an indispensable tool for browsing informatio...

Harvest – An Open Source Toolkit for Extracting Posts and Post Metadata from Web Forums

Automatic extraction of forum posts and metadata is a crucial but challe...

Evolution of a Web-Scale Near Duplicate Image Detection System

Detecting near duplicate images is fundamental to the content ecosystem ...

Boilerplate Removal using a Neural Sequence Labeling Model

The extraction of main content from web pages is an important task for n...

Multi-reference Cosine: A New Approach to Text Similarity Measurement in Large Collections

The importance of an efficient and scalable document similarity detectio...

Similarity Assessment through blocking and affordance assignment in Textual CBR

It has been conceived that children learn new objects through their affo...

Extraction of Relevant Images for Boilerplate Removal in Web Browsers

Boilerplate refers to unwanted and repeated parts of a webpage (such as ...