Systematic Literature Reviews in Software Engineering – Enhancement of the Study Selection Process using Cohen's Kappa Statistic

by   Jorge Pérez, et al.

Context: Systematic literature reviews (SLRs) rely on a rigorous and auditable methodology for minimizing biases and ensuring reliability. A common kind of bias arises when selecting studies using a set of inclusion/exclusion criteria. This bias can be decreased through dual revision, which makes the selection process more time-consuming and remains prone to generating bias depending on how each researcher interprets the inclusion/exclusion criteria. Objective: To reduce the bias and time spent in the study selection process, this paper presents a process for selecting studies based on the use of Cohen's Kappa statistic. We have defined an iterative process based on the use of this statistic during which the criteria are refined until obtain almost perfect agreement (k>0.8). At this point, the two researchers interpret the selection criteria in the same way, and thus, the bias is reduced. Starting from this agreement, dual review can be eliminated; consequently, the time spent is drastically shortened. Method: The feasibility of this iterative process for selecting studies is demonstrated through a tertiary study in the area of software engineering on works that were published from 2005 to 2018. Results: The time saved in the study selection process was 28 the number of studies is sufficiently large, the time saved tend asymptotically to 50 iterative process for selecting studies when conducting SLRs to reduce bias in the interpretation of inclusion and exclusion criteria. It is especially useful for research with few resources.


page 1

page 2

page 3

page 4


Applying Inter-rater Reliability and Agreement in Grounded Theory Studies in Software Engineering

In recent years, the qualitative research on empirical software engineer...

Paving the Way for Mature Secondary Research: The Seven Types of Literature Review

Confusion over different kinds of secondary research, and their divergen...

Using Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews

Background: A systematic literature review (SLR) is a methodology used t...

Towards Continuous Systematic Literature Review in Software Engineering

Context: New scientific evidence continuously arises with advances in So...

Entrofy Your Cohort: A Data Science Approach to Candidate Selection

Selecting a cohort from a set of candidates is a common task within and ...

Reducing the Effort for Systematic Reviews in Software Engineering

Context. Systematic Reviews (SRs) are means for collecting and synthesiz...

Please sign up or login with your details

Forgot password? Click here to reset