University of Oulu

Falessi, D., Juristo, N., Wohlin, C. et al. Empirical software engineering experts on the use of students and professionals in experiments. Empir Software Eng 23, 452–489 (2018).

Empirical software engineering experts on the use of students and professionals in experiments

Saved in:
Author: Falessi, Davide1; Juristo, Natalia2; Wohlin, Claes3;
Organizations: 1California Polytechnic State University, CA, USA
2Universidad Politécnica de Madrid, Spain
3Blekinge Institute of Technology, Sweden
4Brunel University London, UK
5University of Helsinki, Finland
6Reutlingen University, Germany
7Fraunhofer Institute for Experimental Software Engineering, Germany
8University of Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.5 MB)
Persistent link:
Language: English
Published: Springer Nature, 2018
Publish Date: 2020-04-09


Context: Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not.

Objective We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments.

Method We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts’ opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers’ opinions and used the findings to further discuss the statements.

Results Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms.

Conclusion Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.

see all

Series: Empirical software engineering
ISSN: 1382-3256
ISSN-E: 1573-7616
ISSN-L: 1382-3256
Volume: 23
Issue: 1
Pages: 452 - 489
DOI: 10.1007/s10664-017-9523-3
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This research is supported in part by the Academy of Finland Project 278354.
Academy of Finland Grant Number: 278354
Detailed Information: 278354 (Academy of Finland Funding decision)
Copyright information: © Springer Science+Business Media New York 2017. This is a post-peer-review, pre-copyedit version of an article published in Empirical Software Engineering. The final authenticated version is available online at: