McSweeney, Terence P.; Tiulpin, Aleksei; Saarakkala, Simo; Niinimäki, Jaakko; Windsor, Rhydian; Jamaludin, Amir; Kadir, Timor; Karppinen, Jaro; Määttä, Juhani. External Validation of SpineNet, an Open-Source Deep Learning Model for Grading Lumbar Disk Degeneration MRI Features, Using the Northern Finland Birth Cohort 1966. Spine 48(7):p 484-491, April 1, 2023. | DOI: 10.1097/BRS.0000000000004572
External validation of SpineNet, an open-source deep learning model for grading lumbar disk degeneration MRI features, using the Northern Finland Birth Cohort 1966
|Author:||McSweeney, Terence P.1; Tiulpin, Aleksei1,2; Saarakkala, Simo1,2,3;|
1Research Unit of Health Sciences and Technology, University of Oulu
2Finnish Institute of Occupational Health
3Department of Diagnostic Radiology, Oulu University Hospital, Oulu, Finland
4Department of Engineering Science, University of Oxford, UK
5Plexalis Ltd, Oxford, UK
6Rehabilitation Services of South Karelia Social and Health Care District, Lappeenranta
7Medical Research Center Oulu, Oulu University Hospital and University of Oulu, Finland
|Online Access:||PDF Full Text (PDF, 0.9 MB)|
|Persistent link:|| http://urn.fi/urn:nbn:fi-fe20230929137848
|Publish Date:|| 2023-09-29
Study Design. This is a retrospective observational study to externally validate a deep learning image classification model.
Objective. Deep learning models such as SpineNet offer the possibility of automating the process of disk degeneration (DD) classification from magnetic resonance imaging (MRI). External validation is an essential step to their development. The aim of this study was to externally validate SpineNet predictions for DD using Pfirrmann classification and Modic changes (MCs) on data from the Northern Finland Birth Cohort 1966 (NFBC1966).
Summary of Data. We validated SpineNet using data from 1331 NFBC1966 participants for whom both lumbar spine MRI data and consensus DD gradings were available.
Materials and Methods. SpineNet returned Pfirrmann grade and MC presence from T2-weighted sagittal lumbar MRI sequences from NFBC1966, a data set geographically and temporally separated from its training data set. A range of agreement and reliability metrics were used to compare predictions with expert radiologists. Subsets of data that match SpineNet training data more closely were also tested.
Results. Balanced accuracy for DD was 78% (77%–79%) and for MC 86% (85%–86%). Interrater reliability for Pfirrmann grading was Lin concordance correlation coefficient=0.86 (0.85–0.87) and Cohen κ=0.68 (0.67–0.69). In a low back pain subset, these reliability metrics remained largely unchanged. In total, 20.83% of disks were rated differently by SpineNet compared with the human raters, but only 0.85% of disks had a grade difference >1. Interrater reliability for MC detection was κ=0.74 (0.72–0.75). In the low back pain subset, this metric was almost unchanged at κ=0.76 (0.73–0.79).
Conclusions. In this study, SpineNet has been benchmarked against expert human raters in the research setting. It has matched human reliability and demonstrates robust performance despite the multiple challenges facing model generalizability.
|Pages:||484 - 491|
|Type of Publication:||
A1 Journal article – refereed
|Field of Science:||
217 Medical engineering
This research is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 955735. This research was also supported by the National Institute of Arthritis and Musculoskeletal and Skin Disease.
|EU Grant Number:||
(955735) Disc4All - Training network to advance integrated computational simulations in translational medicine, applied to intervertebral disc degeneration
© 2023 The Author(s). Published by Wolters Kluwer Health, Inc. This is an open access article distributed under the Creative Commons Attribution License 4.0 (CCBY), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. http://creativecommons.org/licenses/by/4.0/.