Understanding and Improving Artifact Sharing in Software Engineering Research

by   Christopher S. Timperley, et al.

In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, which include tools, benchmarks, data, and more, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. This often takes the form of a link in the paper pointing to a website containing these additional materials. However, in practice, artifacts suffer from a variety of issues that prevent them from fully realising that potential. To help the software engineering community realise the potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a publication analysis and online survey of 153 software engineering researchers. We apply the established theory of diffusion of innovation, and draw from the field of implementation science, to make evidence-based recommendations. By analysing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using diffusion of innovation as a framework, we analyse how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, using principles from implementation science, we make evidence-based recommendations for specific sub-communities (e.g., students and postdocs, artifact evaluation committees, funding bodies, and professional organisations) to improve the quality of artifacts.


page 1

page 2

page 3

page 4


Treat societally impactful scientific insights as open-source software artifacts

So far, the relationship between open science and software engineering e...

Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing

Research artifact sharing is known to strengthen the transparency of sci...

Motivational models for validating agile requirements in Software Engineering subjects

This paper describes how motivational models can be used to cross check ...

Quality Guidelines for Research Artifacts in Model-Driven Engineering

Sharing research artifacts is known to help people to build upon existin...

Consensus in Software Engineering: A Cognitive Mapping Study

Background: Philosophers of science including Collins, Feyerabend, Kuhn ...

VERIFAS: A Practical Verifier for Artifact Systems

Data-driven workflows, of which IBM's Business Artifacts are a prime exp...

Mining Software Repositories with a Collaborative Heuristic Repository

Many software engineering studies or tasks rely on categorizing software...

Please sign up or login with your details

Forgot password? Click here to reset