University of Oulu

A. Santos, S. Vegas, M. Oivo and N. Juristo, "A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications," in IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1742-1763, 1 Sept. 2021, doi: 10.1109/TSE.2019.2935720

A procedure and guidelines for analyzing groups of software engineering replications

Saved in:
Author: Santos, Adrian1; Vegas, Sira2; Oivo, Markku1;
Organizations: 1M3S (M-Group), ITEE University of Oulu, P.O. Box 3000, 90014, Oulu, Finland
2Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Campus de Montegancedo s/n, 28660 Boadilla del Monte, Madrid, Spain
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.1 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2019
Publish Date: 2019-12-09


Context: Researchers from different groups and institutions are collaborating on building groups of experiments by means of replication (i.e., conducting groups of replications). Disparate aggregation techniques are being applied to analyze groups of replications. The application of unsuitable techniques to aggregate replication results may undermine the potential of groups of replications to provide in-depth insights from experiment results.

Objectives: Provide an analysis procedure with a set of embedded guidelines to aggregate software engineering (SE) replication results.

Method: We compare the characteristics of groups of replications for SE and other mature experimental disciplines such as medicine and pharmacology. In view of their differences, the limitations with regard to the joint data analysis of groups of SE replications and the guidelines provided in mature experimental disciplines to analyze groups of replications, we build an analysis procedure with a set of embedded guidelines specifically tailored to the analysis of groups of SE replications. We apply the proposed analysis procedure to a representative group of SE replications to illustrate its use.

Results: All the information contained within the raw data should be leveraged during the aggregation of replication results. The analysis procedure that we propose encourages the use of stratified individual participant data and aggregated data in tandem to analyze groups of SE replications.

Conclusion: The aggregation techniques used to analyze groups of replications should be justified in research articles. This will increase the reliability and transparency of joint results. The proposed guidelines should ease this endeavor.

see all

Series: IEEE transactions on software engineering
ISSN: 0098-5589
ISSN-E: 1939-3520
ISSN-L: 0098-5589
Volume: 47
Issue: 9
Pages: 1742 - 1763
DOI: 10.1109/TSE.2019.2935720
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was partially funded by Spanish Ministry of Science, Innovation and Universities research grant PGC2018-097265-B-I00.
Copyright information: © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.