Replication and Verifiability in Requirements Engineering: the NLP for RE Case

04/20/2023
by   Sallam Abualhaija, et al.
0

[Context] Study replication is essential for theory building and empirical validation. [Problem] Despite its empirical vocation, requirements engineering (RE) research has given limited attention to study replication, threatening thereby the ability to verify existing results and use previous research as a baseline. [Solution] In this perspective paper, we – a group of experts in natural language processing (NLP) for RE – reflect on the challenges for study replication in NLP for RE. Concretely: (i) we report on hands-on experiences of replication, (ii) we review the state-of-the-art and extract replication-relevant information, and (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction. NLP for RE is a research area that is suitable for study replication since it builds on automated tools which can be shared, and quantitative evaluation that enable direct comparisons between results. [Results] Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks' inherent hairiness, and, in turn, the heterogeneous reporting structure. To address these issues, we propose an ID card whose goal is to provide a structured summary of research papers, with an emphasis on replication-relevant information. [Contribution] We contribute in this study with: (i) a set of reflections on replication in NLP for RE, (ii) a set of recommendations for researchers in the field to increase their awareness on the topic, and (iii) an ID card that is intended to primarily foster replication, and can also be used in other contexts, e.g., for educational purposes. Practitioners will also benefit from the results since replications increase confidence on research findings.

READ FULL TEXT

page 13

page 14

page 16

page 25

page 26

research
06/27/2022

A Model-Based Approach for Specifying Changes in Replications of Empirical Studies in Computer Science

Context: The need of replicating empirical studies in Computer Science (...
research
04/02/2020

Natural Language Processing (NLP) for Requirements Engineering: A Systematic Mapping Study

Natural language processing supported requirements engineering is an are...
research
02/13/2018

Replication studies considered harmful

CONTEXT: There is growing interest in establishing software engineering ...
research
06/13/2018

Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining

The use of machine learning techniques has expanded in education researc...
research
07/14/2022

Node Replication: Theory And Practice

We define and study a term calculus implementing higher-order node repli...
research
03/10/2021

Financial factors selection with knockoffs: fund replication, explanatory and prediction networks

We apply the knockoff procedure to factor selection in finance. By build...

Please sign up or login with your details

Forgot password? Click here to reset