An Empirically Evaluated Checklist for Surveys in Software Engineering

by   Jefferson Seide Molléri, et al.

Context: Over the past decade Software Engineering research has seen a steady increase in survey-based studies, and there are several guidelines providing support for those willing to carry out surveys. The need for auditing survey research has been raised in the literature. Checklists have been used to assess different types of empirical studies, such as experiments and case studies. Objective: This paper proposes a checklist to support the design and assessment of survey-based research in software engineering grounded in existing guidelines for survey research. We further evaluated the checklist in the research practice context. Method: To construct the checklist, we systematically aggregated knowledge from 14 methodological papers supporting survey-based research in software engineering. We identified the key stages of the survey process and its recommended practices through thematic analysis and vote counting. To improve our initially designed checklist we evaluated it using a mixed evaluation approach involving experienced researchers. Results: The evaluation provided insights regarding limitations of the checklist in relation to its understanding and objectivity. In particular, 19 of the 38 checklist items were improved according to the feedback received from its evaluation. Finally, a discussion on how to use the checklist and what its implications are for research practice is also provided. Conclusion: The proposed checklist is an instrument suitable for auditing survey reports as well as a support tool to guide ongoing research with regard to the survey design process.


page 1

page 2

page 3

page 4


Case Survey Studies in Software Engineering Research

Background: Given the social aspects of Software Engineering (SE), in th...

Challenges in Survey Research

While being an important and often used research method, survey research...

Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys

Requirements Engineering (RE) has established itself as a software engin...

A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering

Context: Systematic literature reviews (SLRs) have become standard pract...

Pitfalls and Guidelines for Using Time-Based Git Data

Many software engineering research papers rely on time-based data (e.g.,...

Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing

Research artifact sharing is known to strengthen the transparency of sci...

Optimization in Software Engineering – A Pragmatic Approach

Empirical software engineering is concerned with the design and analysis...

Please sign up or login with your details

Forgot password? Click here to reset