‘Standing on the shoulders of giants’ is what we say to highlight that scientific progress is heavily relying on previous discoveries. Oftentimes we are willing to build on top of these results but have a hard time to continue the work because the results in the original publication are difficult to reproduce. A recent survey in Nature  shows that, in a pool of 1,576 researchers, more than 70% failed to reproduce another researcher’s experiment and—even more telling—more than half of the survey participants could not reproduce their own experiments. The reasons for these results are having to do with selective reporting, cherry picking of experimental results, and lack of information about the experiments, to name a few.
While the aforementioned survey has participants mostly from natural sciences, one might think that the situation in computer science is different. But it is actually not! Not long ago, Collberg et al.  reported that only a third of the code and data, in the 13 venues they selected, are available and can be easily built within 30 minutes. And they have not even tried to reproduce the experiments reported in the respective papers. While reproducibility of computational research could be challenging (due to specialized hardware, complicated setups and library dependencies, etc.), reproducing empirical studies, such as observing users to interact with a particular system, is often out of the question.
It is our belief that sharing experimental code, data and setup will benefit scientific progress, foster collaboration and exchange of ideas. We would like to build a culture where sharing results, code, and scripts is the norm rather than an exception. Since we recognize the additional effort, we aim to build technical expertise on how to do this efficiently and conduct better research via creating repeatable and shareable methods and results.
Hence, as an author of a research track paper, we would like to invite you to submit your contribution to the ISWC 2020 Reproducibility Track (In this call the term reproducibility refers to the case when an independent researcher is trying to re-run the same experimental setup and reproduce the most important results of the paper. As there are terminological issues, discussion and comparison of terminology can be found here .).
The ISWC Reproducibility Initiative has the following goals:
- To enable easy sharing of code and experimental set-ups (take a paper and reuse it).
- To make more code and data available.
- To highlight the impact and increase the credibility of the Semantic Web research.
- To facilitate the dissemination of research results
Why should I be a part of this?
- To easily repeat your own experiments.
- To discover accidental flaws and improve your results.
- To increase confidence in your results.
- To make it easy for other researchers to compare to, adopt and extend your research.
- To increase visibility and impact of your results.
For the reproducibility assessment, two independent members of the Programme Committee will interact with the authors to check the availability of the data, source code, documentation, configuration requirements and reproduce the most important results of the paper.
Submission: via EasyChair
Submission guidelines will be added later.
Submission Deadline: July 10, 2020
Assessment Period: 17 July – 17 September
Results Announcement (During ISWC 2020): Nov 2-6, 2020
Reproducibility Track Chairs:
- Valentina Ivanova, RISE Research Institutes of Sweden
- Pasquale Minervini ,University College London, United Kingdom
Acknowledgements: The text of this call is partially based on the call for the first edition of the Reproducibility Initiative at ISWC 2019 (https://iswc2019.semanticweb.org/call-for-reproducibility/) by Alejandra Gonzalez-Beltran & Michael Cochez.
 Is there a reproducibility crisis in science?, M. Baker, Nature 533, 452–454 (2016), https://doi.org/10.1038/533452
 Repeatability and Benefaction in Computer Systems Research, C. Collberg, T. Proebsting, A. M. Warren, University of Arizona TR 14-04, 2015.
 Reproducibility vs. Replicability: A Brief History of a Confused Terminology, H. E. Plesser, Frontiers in neuroinformatics 11 (2018): 76, https://doi.org/10.3389/fninf.2017.00076