Understanding and Improving Artifact Sharing in Software Engineering Research

by   Christopher S. Timperley, et al.

In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, which include tools, benchmarks, data, and more, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. This often takes the form of a link in the paper pointing to a website containing these additional materials. However, in practice, artifacts suffer from a variety of issues that prevent them from fully realising that potential. To help the software engineering community realise the potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a publication analysis and online survey of 153 software engineering researchers. We apply the established theory of diffusion of innovation, and draw from the field of implementation science, to make evidence-based recommendations. By analysing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using diffusion of innovation as a framework, we analyse how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, using principles from implementation science, we make evidence-based recommendations for specific sub-communities (e.g., students and postdocs, artifact evaluation committees, funding bodies, and professional organisations) to improve the quality of artifacts.


page 1

page 2

page 3

page 4


The Dangerous Dogmas of Software Engineering

To legitimize itself as a scientific discipline, the software engineerin...

Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing

Research artifact sharing is known to strengthen the transparency of sci...

Quality Guidelines for Research Artifacts in Model-Driven Engineering

Sharing research artifacts is known to help people to build upon existin...

Science-Software Linkage: The Challenges of Traceability between Scientific Knowledge and Software Artifacts

Although computer science papers are often accompanied by software artif...

VERIFAS: A Practical Verifier for Artifact Systems

Data-driven workflows, of which IBM's Business Artifacts are a prime exp...

Grand Challenges of Traceability: The Next Ten Years

In 2007, the software and systems traceability community met at the firs...

Consensus in Software Engineering: A Cognitive Mapping Study

Background: Philosophers of science including Collins, Feyerabend, Kuhn ...