Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study

01/19/2022
by   Wojciech Kusa, et al.
0

In the process of Systematic Literature Review, citation screening is estimated to be one of the most time-consuming steps. Multiple approaches to automate it using various machine learning techniques have been proposed. The first research papers that apply deep neural networks to this problem were published in the last two years. In this work, we conduct a replicability study of the first two deep learning papers for citation screening and evaluate their performance on 23 publicly available datasets. While we succeeded in replicating the results of one of the papers, we were unable to replicate the results of the other. We summarise the challenges involved in the replication, including difficulties in obtaining the datasets to match the experimental setup of the original papers and problems with executing the original source code. Motivated by this experience, we subsequently present a simpler model based on averaging word embeddings that outperforms one of the models on 18 out of 23 datasets and is, on average, 72 times faster than the second replicated approach. Finally, we measure the training time and the invariance of the models when exposed to a variety of input features and random initialisations, demonstrating differences in the robustness of these approaches.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/04/2023

CRUISE-Screening: Living Literature Reviews Toolbox

Keeping up with research and finding related work is still a time-consum...
research
02/21/2022

An open-source integrated framework for the automation of citation collection and screening in systematic reviews

The exponential growth of scientific production makes secondary literatu...
research
11/19/2020

From Protocol to Screening: A Hybrid Learning Approach for Technology-Assisted Systematic Literature Reviews

In the medical domain, a Systematic Literature Review (SLR) attempts to ...
research
08/10/2020

Measure the Impact of Institution and Paper via Institution-citation Network

This paper investigates the impact of institutes and papers over time ba...
research
03/21/2018

Crowd-based Multi-Predicate Screening of Papers in Literature Reviews

Systematic literature reviews (SLRs) are one of the most common and usef...
research
08/12/2023

Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation

Medical systematic reviews can be very costly and resource intensive. We...
research
03/31/2019

SysML'19 demo: customizable and reusable Collective Knowledge pipelines to automate and reproduce machine learning experiments

Reproducing, comparing and reusing results from machine learning and sys...

Please sign up or login with your details

Forgot password? Click here to reset