Understanding and Improving Artifact Sharing in Software Engineering Research
share
In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, which include tools, benchmarks, data, and more, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. This often takes the form of a link in the paper pointing to a website containing these additional materials. However, in practice, artifacts suffer from a variety of issues that prevent them from fully realising that potential. To help the software engineering community realise the potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a publication analysis and online survey of 153 software engineering researchers. We apply the established theory of diffusion of innovation, and draw from the field of implementation science, to make evidence-based recommendations. By analysing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using diffusion of innovation as a framework, we analyse how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, using principles from implementation science, we make evidence-based recommendations for specific sub-communities (e.g., students and postdocs, artifact evaluation committees, funding bodies, and professional organisations) to improve the quality of artifacts.
READ FULL TEXT