H. H. Nguyen, S. Saarakkala, M. B. Blaschko and A. Tiulpin, "CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting," 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2022, pp. 1-5, doi: 10.1109/ISBI52829.2022.9761545.
CLIMAT : clinically-inspired multi-agent transformers for knee osteoarthritis trajectory forecasting
|Author:||Nguyen, Huy Hoang1; Saarakkala, Simo1,2; Blaschko, Matthew B.3;|
1University of Oulu, Finland
2Oulu University Hospital, Finland
3KU Leuven, Belgium
|Online Access:||PDF Full Text (PDF, 2.6 MB)|
|Persistent link:|| http://urn.fi/urn:nbn:fi-fe2022101361858
Institute of Electrical and Electronics Engineers,
|Publish Date:|| 2022-10-13
In medical applications, deep learning methods are designed to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents — a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at https://github.com/MIPT-Oulu/CLIMAT.
IEEE International Symposium on Biomedical Imaging
2022 IEEE 19th international symposium on biomedical imaging (ISBI)
|Type of Publication:||
A4 Article in conference proceedings
|Field of Science:||
217 Medical engineering
3126 Surgery, anesthesiology, intensive care, radiology
The OAI is a public-private partnership comprised of five contracts (N01- AR-2-2258; N01-AR-2-2259; N01-AR-2- 2260; N01-AR-2-2261; N01-AR-2-2262) funded by the National Institutes of Health, a branch of the Department of Health and Human Services, and conducted by the OAI Study Investigators. Private funding partners include Merck Research Laboratories; Novartis Pharmaceuticals Corporation, GlaxoSmithKline; and Pfizer, Inc. Private sector funding for the OAI is managed by the Foundation for the National Institutes of Health. The authors wish to acknowledge CSC – IT Center for Science, Finland, for generous computational resources. We would like to acknowledge the strategic funding of the University of Oulu, Sigrid Juselius Foundation, Finland.
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.