COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews

08/09/2021
by   Shruti Singh, et al.
8

Comparing research papers is a conventional method to demonstrate progress in experimental research. We present COMPARE, a taxonomy and a dataset of comparison discussions in peer reviews of research papers in the domain of experimental deep learning. From a thorough observation of a large set of review sentences, we build a taxonomy of categories in comparison discussions and present a detailed annotation scheme to analyze this. Overall, we annotate 117 reviews covering 1,800 sentences. We experiment with various methods to identify comparison sentences in peer reviews and report a maximum F1 Score of 0.49. We also pretrain two language models specifically on ML, NLP, and CV paper abstracts and reviews to learn informative representations of peer reviews. The annotated dataset and the pretrained models are available at https://github.com/shruti-singh/COMPARE .

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/25/2018

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Peer reviewing is a central component in the scientific publishing proce...
research
07/11/2016

Qualitative Judgement of Research Impact: Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research

The appeal of metric evaluation of research impact has attracted conside...
research
01/30/2021

Can We Automate Scientific Reviewing?

The rapid development of science and technology has been accompanied by ...
research
07/20/2022

Journal Impact Factor and Peer Review Thoroughness and Helpfulness: A Supervised Machine Learning Study

The journal impact factor (JIF) is often equated with journal quality an...
research
10/16/2021

A Dataset for Discourse Structure in Peer Review Discussions

At the foundation of scientific evaluation is the labor-intensive proces...
research
03/27/2019

Does My Rebuttal Matter? Insights from a Major NLP Conference

Peer review is a core element of the scientific process, particularly in...
research
03/25/2019

Argument Mining for Understanding Peer Reviews

Peer-review plays a critical role in the scientific writing and publicat...

Please sign up or login with your details

Forgot password? Click here to reset