University of Oulu

C. Ayala, B. Turhan, X. Franch and N. Juristo, "Use and Misuse of the Term Experiment in Mining Software Repositories Research," in IEEE Transactions on Software Engineering, doi: 10.1109/TSE.2021.3113558

Use and Misuse of the Term Experiment in Mining Software Repositories Research

Saved in:
Author: Ayala, Claudia1; Turhan, Burak2,3; Franch, Xavier1;
Organizations: 1Universitat Politècnica de Catalunya, BarcelonaTECH, Campus Nord - Jordi Girona 1-3 Barcelona, Spain. CO 08034
2University of Oulu, Pentti Kaiteran katu 1, Linnanmaa, Finland
3Monash University
4Universidad Politécnica de Madrid, Campus de Montegancedo. Boadilla del Monte. Spain. 28660
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.5 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2021-10-06


The significant momentum and importance of Mining Software Repositories (MSR) in Software Engineering (SE) has fostered new opportunities and challenges for extensive empirical research. However, MSR researchers seem to struggle to characterize the empirical methods they use into the existing empirical SE body of knowledge. This is especially the case of MSR experiments. To provide evidence on the special characteristics of MSR experiments and their differences with experiments traditionally acknowledged in SE so far, we elicited the hallmarks that differentiate an experiment from other types of empirical studies and characterized the hallmarks and types of experiments in MSR. We analyzed MSR literature obtained from a small-scale systematic mapping study to assess the use of the term experiment in MSR. We found that 19% of the papers claiming to be an experiment are indeed not an experiment at all but also observational studies, so they use the term in a misleading way. From the remaining 81% of the papers, only one of them refers to a genuine controlled experiment while the others stand for experiments with limited control. MSR researchers tend to overlook such limitations, compromising the interpretation of the results of their studies. We provide recommendations and insights to support the improvement of MSR experiments.

see all

Series: IEEE transactions on software engineering
ISSN: 0098-5589
ISSN-E: 1939-3520
ISSN-L: 0098-5589
Volume: Early Access
Issue: Early Access
Pages: 1 - 20
DOI: 10.1109/TSE.2021.3113558
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: We are extremely grateful to Dr. Xin Xia for their valuable and constructive feedback.
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.