Evaluation and Comparison of Open Source Bibliographic Reference Parsers: A Business Use Case

02/04/2018
by   Dominika Tkaczyk, et al.
0

Bibliographic reference parsing refers to extracting machine-readable metadata, such as the names of the authors, the title, or journal name, from bibliographic reference strings. Many approaches to this problem have been proposed so far, including regular expressions, knowledge bases and supervised machine learning. Many open source reference parsers based on various algorithms are also available. In this paper, we apply, evaluate and compare ten reference parsing tools in a specific business use case. The tools are Anystyle-Parser, Biblio, CERMINE, Citation, Citation-Parser, GROBID, ParsCit, PDFSSA4MET, Reference Tagger and Science Parse, and we compare them in both their out-of-the-box versions and tuned to the project-specific data. According to our evaluation, the best performing out-of-the-box tool is GROBID (F1 0.89), followed by CERMINE (F1 0.83) and ParsCit (F1 0.75). We also found that even though machine learning-based tools and tools based on rules or regular expressions achieve on average similar precision (0.77 for ML-based tools vs. 0.76 for non-ML-based tools), applying machine learning-based tools results in the recall three times higher than in the case of non-ML-based tools (0.66 vs. 0.22). Our study also confirms that tuning the models to the task-specific data results in the increase in the quality. The retrained versions of reference parsers are in all cases better than their out-of-the-box counterparts; for GROBID F1 increased by 3 and for ParsCit by 16

READ FULL TEXT

page 7

page 8

research
02/04/2018

Machine Learning vs. Rules and Out-of-the-Box vs. Retrained: An Evaluation of Open-Source Bibliographic Reference and Citation Parsers

Bibliographic reference parsing refers to extracting machine-readable me...
research
11/26/2018

ParsRec: A Novel Meta-Learning Approach to Recommending Bibliographic Reference Parsers

Bibliographic reference parsers extract machine-readable metadata such a...
research
08/27/2018

ParsRec: Meta-Learning Recommendations for Bibliographic Reference Parsing

Bibliographic reference parsers extract metadata (e.g. author names, tit...
research
06/09/2020

Using BibTeX to Automatically Generate Labeled Data for Citation Field Extraction

Accurate parsing of citation reference strings is crucial to automatical...
research
05/29/2022

Structured references from PDF articles: assessing the tools for bibliographic reference extraction and parsing

Many solutions have been provided to extract bibliographic references fr...
research
05/12/2018

Citation Data-set for Machine Learning Citation Styles and Entity Extraction from Citation Strings

Citation parsing is fundamental for search engines within academia and t...

Please sign up or login with your details

Forgot password? Click here to reset