University of Oulu

Simone Romano, Davide Fucci, Giuseppe Scanniello, Maria Teresa Baldassarre, Burak Turhan, Natalia Juristo, On researcher bias in Software Engineering experiments, Journal of Systems and Software, Volume 182, 2021, 111068, ISSN 0164-1212,

On researcher bias in Software Engineering experiments

Saved in:
Author: Romano, Simone1; Fucci, Davide2; Scanniello, Giuseppe3;
Organizations: 1University of Bari, Bari, Italy
2Blekinge Institute of Technology, Karlskrona, Sweden
3University of Basilicata, Potenza, Italy
4University of Oulu, Oulu, Finland
5Monash University, Melbourne, Australia
6Universidad Politécnica de Madrid, Madrid, Spain
Format: article
Version: accepted version
Access: embargoed
Persistent link:
Language: English
Published: Elsevier, 2021
Publish Date: 2023-08-27


Researcher bias occurs when researchers influence the results of an empirical study based on their expectations, either consciously or unconsciously. Researcher bias might be due to the use of Questionable Research Practices (QRPs). In research fields like medicine, blinding techniques have been applied to counteract researcher bias. In this paper, we present two studies to increase our body of knowledge on researcher bias in Software Engineering (SE) experiments, including: (i) QRPs potentially leading to researcher bias; (ii) causes behind researcher bias; and (iii) possible actions to counteract researcher bias with a focus on, but not limited to, blinding techniques. The former is an interview study, intended as an exploratory study, with nine experts of the empirical SE community. The latter is a quantitative survey with 51 respondents, who were experts of the above-mentioned community. The findings from the exploratory study represented the starting point to design the survey. In particular, we defined the questionnaire of this survey to support the findings from the exploratory study. From the interview study, it emerged that some QRPs (e.g., post-hoc outlier criteria) are acceptable in certain cases. Also, it appears that researcher bias is perceived in SE and, to counteract researcher bias, a number of solutions have been highlighted. For example, duplicating the data analysis in SE experiments or fostering open data policies in SE conferences/journals. The findings from the interview study are mostly confirmed by those from the survey, and allowed us to delineate recommendations to counteract researcher bias in SE experiments. Some recommendations are intended for SE researchers, while others are purposeful for the boards of SE research venues.

see all

Series: Journal of systems and software
ISSN: 0164-1212
ISSN-E: 1873-1228
ISSN-L: 0164-1212
Volume: 182
Article number: 111068
DOI: 10.1016/j.jss.2021.111068
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: The authors would like to thank both interviewees and respondents for their participation in the studies presented in this paper.
Copyright information: © 2021 Elsevier Inc. All rights reserved. This manuscript version is made available under the CC-BY-NC-ND 4.0 license