Publish or perish, but do not forget your software artifacts
- PDF / 1,652,669 Bytes
- 32 Pages / 439.642 x 666.49 pts Page_size
- 18 Downloads / 206 Views
Publish or perish, but do not forget your software artifacts 1 ¨ Robert Heumuller 1 Frank Ortmeier
· Sebastian Nielebock1
1,2 ¨ · Jacob Kruger
·
Published online: 8 October 2020 © The Author(s) 2020
Abstract Open-science initiatives have gained substantial momentum in computer science, and particularly in software-engineering research. A critical aspect of open-science is the public availability of artifacts (e.g., tools), which facilitates the replication, reproduction, extension, and verification of results. While we experienced that many artifacts are not publicly available, we are not aware of empirical evidence supporting this subjective claim. In this article, we report an empirical study on software artifact papers (SAPs) published at the International Conference on Software Engineering (ICSE), in which we investigated whether and how researchers have published their software artifacts, and whether this had scientific impact. Our dataset comprises 789 ICSE research track papers, including 604 SAPs (76.6 %), from the years 2007 to 2017. While showing a positive trend towards artifact availability, our results are still sobering. Even in 2017, only 58.5 % of the papers that stated to have developed a software artifact made that artifact publicly available. As we did find a small, but statistically significant, positive correlation between linking to artifacts in a paper and its scientific impact in terms of citations, we hope to motivate the research community to share more artifacts. With our insights, we aim to support the advancement of open science by discussing our results in the context of existing initiatives and guidelines. In particular, our findings advocate the need for clearly communicating artifacts and the use of non-commercial, persistent archives to provide replication packages. Keywords Software · Open science · Open source · Artifacts · Publishing
Communicated by: Martin Monperrus This paper has been awarded the Empirical Software Engineering (EMSE) open science badge. The work of Jacob Kr¨uger has been supported by the German Research Foundation (SA 465/49-3) and an IFI fellowship of the German Academic Exchange Service. Robert Heum¨uller and Sebastian Nielebock contributed equally to the research reported in this article. Robert Heum¨uller
[email protected]
Extended author information available on the last page of the article.
4586
Empirical Software Engineering (2020) 25:4585–4616
1 Introduction Software-engineering research has always been driven by developing concepts and techniques to automate or facilitate the tasks of software developers (Wicks and Dewar 2007; Ossher et al. 2000). As a consequence, researchers have been building numerous software artifacts, ranging from analysis scripts for empirical studies, over prototypes to show the feasibility of a technique, to full-fledged tools that are used in practice. In the context of this article, software artifacts represent runnable (at least after compiling) programs, developed by researchers to obtain or analyze th
Data Loading...
