Casey, A. E., Ansari, S., Nakisa, B., Kelly, B., Brown, P., Cooper, P., Muhammad, I., Livingstone, S., Reddy, S., & Makinen, V.-P. (2023). Application of a comprehensive evaluation framework to covid-19 studies: Systematic review of translational aspects of artificial intelligence in health care. JMIR AI, 2, e42313. https://doi.org/10.2196/42313
Application of a comprehensive evaluation framework to COVID-19 studies : systematic review of translational aspects of artificial intelligence in health care
|Author:||Casey, Aaron Edward1,2; Ansari, Saba3; Nakisa, Bahareh4;|
1South Australian Health and Medical Research Institute, Adelaide, Australia
2Australian Centre for Precision Health, Cancer Research Institute, University of South Australia, Adelaide, Australia
3School of Medicine, Deakin University, Geelong, Australia
4School of Information Technology, Deakin University, Geelong, Australia
5Library, Deakin University, Geelong, Australia
6Orion Health, Auckland, New Zealand
7Computational Medicine, Faculty of Medicine, University of Oulu, Oulu, Finland
8Centre for Life Course Health Research, Faculty of Medicine, University of Oulu, Oulu, Finland
|Online Access:||PDF Full Text (PDF, 0.6 MB)|
|Persistent link:|| http://urn.fi/urn:nbn:fi-fe20231023140901
|Publish Date:|| 2023-10-23
Background: Despite immense progress in artificial intelligence (AI) models, there has been limited deployment in health care environments. The gap between potential and actual AI applications is likely due to the lack of translatability between controlled research environments (where these models are developed) and clinical environments for which the AI tools are ultimately intended.
Objective: We previously developed the Translational Evaluation of Healthcare AI (TEHAI) framework to assess the translational value of AI models and to support successful transition to health care environments. In this study, we applied the TEHAI framework to the COVID-19 literature in order to assess how well translational topics are covered.
Methods: A systematic literature search for COVID-19 AI studies published between December 2019 and December 2020 resulted in 3830 records. A subset of 102 (2.7%) papers that passed the inclusion criteria was sampled for full review. The papers were assessed for translational value and descriptive data collected by 9 reviewers (each study was assessed by 2 reviewers). Evaluation scores and extracted data were compared by a third reviewer for resolution of discrepancies. The review process was conducted on the Covidence software platform.
Results: We observed a significant trend for studies to attain high scores for technical capability but low scores for the areas essential for clinical translatability. Specific questions regarding external model validation, safety, nonmaleficence, and service adoption received failed scores in most studies.
Conclusions: Using TEHAI, we identified notable gaps in how well translational topics of AI models are covered in the COVID-19 clinical sphere. These gaps in areas crucial for clinical translatability could, and should, be considered already at the model development stage to increase translatability into real COVID-19 health care environments.
|Type of Publication:||
A2 Review article in a scientific journal
|Field of Science:||
3141 Health care science
113 Computer and information sciences
© Aaron Edward Casey, Saba Ansari, Bahareh Nakisa, Blair Kelly, Pieta Brown, Paul Cooper, Imran Muhammad, Steven Livingstone, Sandeep Reddy, Ville-Petteri Makinen. Originally published in JMIR AI (https://ai.jmir.org), 06.07.2023. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR AI, is properly cited. The complete bibliographic information, a link to the original publication on https://www.ai.jmir.org/, as well as this copyright and license information must be included.