University of Oulu

Casey, A. E., Ansari, S., Nakisa, B., Kelly, B., Brown, P., Cooper, P., Muhammad, I., Livingstone, S., Reddy, S., & Makinen, V.-P. (2023). Application of a comprehensive evaluation framework to covid-19 studies: Systematic review of translational aspects of artificial intelligence in health care. JMIR AI, 2, e42313.

Application of a comprehensive evaluation framework to COVID-19 studies : systematic review of translational aspects of artificial intelligence in health care

Saved in:
Author: Casey, Aaron Edward1,2; Ansari, Saba3; Nakisa, Bahareh4;
Organizations: 1South Australian Health and Medical Research Institute, Adelaide, Australia
2Australian Centre for Precision Health, Cancer Research Institute, University of South Australia, Adelaide, Australia
3School of Medicine, Deakin University, Geelong, Australia
4School of Information Technology, Deakin University, Geelong, Australia
5Library, Deakin University, Geelong, Australia
6Orion Health, Auckland, New Zealand
7Computational Medicine, Faculty of Medicine, University of Oulu, Oulu, Finland
8Centre for Life Course Health Research, Faculty of Medicine, University of Oulu, Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 0.6 MB)
Persistent link:
Language: English
Published: JMIR Publications, 2023
Publish Date: 2023-10-23


Background: Despite immense progress in artificial intelligence (AI) models, there has been limited deployment in health care environments. The gap between potential and actual AI applications is likely due to the lack of translatability between controlled research environments (where these models are developed) and clinical environments for which the AI tools are ultimately intended.

Objective: We previously developed the Translational Evaluation of Healthcare AI (TEHAI) framework to assess the translational value of AI models and to support successful transition to health care environments. In this study, we applied the TEHAI framework to the COVID-19 literature in order to assess how well translational topics are covered.

Methods: A systematic literature search for COVID-19 AI studies published between December 2019 and December 2020 resulted in 3830 records. A subset of 102 (2.7%) papers that passed the inclusion criteria was sampled for full review. The papers were assessed for translational value and descriptive data collected by 9 reviewers (each study was assessed by 2 reviewers). Evaluation scores and extracted data were compared by a third reviewer for resolution of discrepancies. The review process was conducted on the Covidence software platform.

Results: We observed a significant trend for studies to attain high scores for technical capability but low scores for the areas essential for clinical translatability. Specific questions regarding external model validation, safety, nonmaleficence, and service adoption received failed scores in most studies.

Conclusions: Using TEHAI, we identified notable gaps in how well translational topics of AI models are covered in the COVID-19 clinical sphere. These gaps in areas crucial for clinical translatability could, and should, be considered already at the model development stage to increase translatability into real COVID-19 health care environments.

see all

Series: JMIR AI
ISSN: 2817-1705
ISSN-E: 2817-1705
ISSN-L: 2817-1705
Volume: 2
Article number: e42313
DOI: 10.2196/42313
Type of Publication: A2 Review article in a scientific journal
Field of Science: 3141 Health care science
113 Computer and information sciences
Copyright information: © Aaron Edward Casey, Saba Ansari, Bahareh Nakisa, Blair Kelly, Pieta Brown, Paul Cooper, Imran Muhammad, Steven Livingstone, Sandeep Reddy, Ville-Petteri Makinen. Originally published in JMIR AI (, 06.07.2023. This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR AI, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.