Modelo de estrategias de reproducibilidad para experimentos en ingeniería de software.

Abstract

In this paper, we explored the reproducibility of experiments in empirical software engineering. Reproducibility has been an important and controvert aspect for scientific knowledge generation. Reproducibility analysis helps the research field and contributes to the generation of reliable knowledge for evidence-based software engineering. Several studies have reported low replication of research, associating this problem with the well-known “reproducibility crisis”. With this consideration, we investigated this context to improve the reproducibility in software experimentation. We use the Design Science methodology for the construction of an artifact. With the help of systematic reviews, interviews, and our own applications, we investigate reproducibility in the context of empirical software engineering systematizing the problem and needs. We contemplate the maturity model Problem-Solving Model to determine the presence and absent of elements within the experimentation process. We formulate a multilevel model of intentions and strategies to describe a supporting process based on quality factors: self-description, transferability, and standardization. Through an experiment, which considered the participation of students with notions about research in software engineering and expert researchers, we validated the effectiveness of the proposal. We carried out several analyzes and determined that: (i) students do not perceive differences in the complexity of using the proposal; (ii) experts tend to assess the relevance of reproducibility factors differently when they are applied; (iii) the effectiveness of the proposal does not generate a significant change; (iv) we found a difference in portability and standardization factors, with a significant difference in the use of repositories and description of packaging.