University of Oulu

Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, Fabio Palomba, A critical comparison on six static analysis tools: Detection, agreement, and precision, Journal of Systems and Software, Volume 198, 2023, 111575, ISSN 0164-1212,

A critical comparison on six static analysis tools : detection, agreement, and precision

Saved in:
Author: Lenarduzzi, Valentina1; Pecorelli, Fabiano2; Saarimaki, Nyyti2;
Organizations: 1M3S Research Unit - University of Oulu, Finland
2Clowee Research group - Tampere University, Finland
3SeSa Lab - University of Salerno, Italy
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 0.5 MB)
Persistent link:
Language: English
Published: Elsevier, 2022
Publish Date: 2023-05-04


Background: Developers use Static Analysis Tools (SATs) to control for potential quality issues in source code, including defects and technical debt. Tool vendors have devised quite a number of tools, which makes it harder for practitioners to select the most suitable one for their needs. To better support developers, researchers have been conducting several studies on SATs to favor the understanding of their actual capabilities.

Aims: Despite the work done so far, there is still a lack of knowledge regarding (1) what is their agreement, and (2) what is the precision of their recommendations. We aim at bridging this gap by proposing a large-scale comparison of six popular SATs for Java projects: Better Code Hub, CheckStyle, Coverity Scan, FindBugs, PMD, and SonarQube.

Methods: We analyze 47 Java projects applying 6 SATs. To assess their agreement, we compared them by manually analyzing – at line – and class-level — whether they identify the same issues. Finally, we evaluate the precision of the tools against a manually-defined ground truth.

Results: The key results show little to no agreement among the tools and a low degree of precision.

Conclusion: Our study provides the first overview on the agreement among different tools as well as an extensive analysis of their precision that can be used by researchers, practitioners, and tool vendors to map the current capabilities of the tools and envision possible improvements.

see all

Series: Journal of systems and software
ISSN: 0164-1212
ISSN-E: 1873-1228
ISSN-L: 0164-1212
Volume: 198
Article number: 111575
DOI: 10.1016/j.jss.2022.111575
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: Fabio is partially supported by the Swiss National Science Foundation - SNF Project No. PZ00P2_186090. Furthermore, this work has been partially supported by the EMELIOT national research project, which has been funded by the MUR under the PRIN 2020 program (Contract No. 2020W3A5FY).
Copyright information: © 2022 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license (