University of Oulu

I. Karac, B. Turhan and N. Juristo, "A Controlled Experiment with Novice Developers on the Impact of Task Description Granularity on Software Quality in Test-Driven Development," in IEEE Transactions on Software Engineering, vol. 47, no. 7, pp. 1315-1330, 1 July 2021, doi: 10.1109/TSE.2019.2920377

A controlled experiment with novice developers on the impact of task description granularity on software quality in test-driven development

Saved in:
Author: Karac, Itir1; Turhan, Burak2; Juristo, Natalia3
Organizations: 1M3S Research Unit, University of Oulu, Finland
2Monash University, Australia
3Escuela Tecnica Superior de Ingenieros Informaticos, Universidad Politecnica de Madrid, Spain
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.5 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2019
Publish Date: 2020-01-08


Background: Test-Driven Development (TDD) is an iterative software development process characterized by test-code-refactor cycle. TDD recommends that developers work on small and manageable tasks at each iteration. However, the ability to break tasks into small work items effectively is a learned skill that improves with experience. In experimental studies of TDD, the granularity of task descriptions is an overlooked factor. In particular, providing a more granular task description in terms of a set of sub-tasks versus providing a coarser-grained, generic description.

Objective: We aim to investigate the impact of task description granularity on the outcome of TDD, as implemented by novice developers, with respect to software quality, as measured by functional correctness and functional completeness.

Method: We conducted a one-factor crossover experiment with 48 graduate students in an academic environment. Each participant applied TDD and implemented two tasks, where one of the tasks was presented using a more granular task description. Resulting artifacts were evaluated with acceptance tests to assess functional correctness and functional completeness. Linear mixed-effects models (LMM) were used for analysis.

Results: Software quality improved significantly when participants applied TDD using more granular task descriptions. The effect of task description granularity is statistically significant and had a medium to large effect size. Moreover, the task was found to be a significant predictor of software quality which is an interesting result (because two tasks used in the experiment were considered to be of similar complexity).

Conclusion: For novice TDD practitioners, the outcome of TDD is highly coupled with the ability to break down the task into smaller parts. For researchers, task selection and task description granularity requires more attention in the design of TDD experiments. Task description granularity should be taken into account in secondary studies. Further comparative studies are needed to investigate whether task descriptions affect other development processes similarly.

see all

Series: IEEE transactions on software engineering
ISSN: 0098-5589
ISSN-E: 1939-3520
ISSN-L: 0098-5589
Volume: 47
Issue: 7
Pages: 1315 - 1330
DOI: 10.1109/TSE.2019.2920377
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was partially funded by Academy of Finland Project 278354 and Spanish Ministry of Science, Innovation and Universities research grant PGC2018-097265-B-I00.
Academy of Finland Grant Number: 278354
Detailed Information: 278354 (Academy of Finland Funding decision)
Copyright information: © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.