University of Oulu

H. H. Nguyen, S. Saarakkala, M. B. Blaschko and A. Tiulpin, "CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting," 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2022, pp. 1-5, doi: 10.1109/ISBI52829.2022.9761545.

CLIMAT : clinically-inspired multi-agent transformers for knee osteoarthritis trajectory forecasting

Saved in:
Author: Nguyen, Huy Hoang1; Saarakkala, Simo1,2; Blaschko, Matthew B.3;
Organizations: 1University of Oulu, Finland
2Oulu University Hospital, Finland
3KU Leuven, Belgium
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 2.6 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2022-10-13


In medical applications, deep learning methods are designed to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents — a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at

see all

Series: IEEE International Symposium on Biomedical Imaging
ISSN: 1945-7928
ISSN-E: 1945-8452
ISSN-L: 1945-7928
ISBN: 978-1-6654-2923-8
ISBN Print: 978-1-6654-2924-5
Article number: 9761545
DOI: 10.1109/isbi52829.2022.9761545
Host publication: 2022 IEEE 19th international symposium on biomedical imaging (ISBI)
Type of Publication: A4 Article in conference proceedings
Field of Science: 217 Medical engineering
3126 Surgery, anesthesiology, intensive care, radiology
Funding: The OAI is a public-private partnership comprised of five contracts (N01- AR-2-2258; N01-AR-2-2259; N01-AR-2- 2260; N01-AR-2-2261; N01-AR-2-2262) funded by the National Institutes of Health, a branch of the Department of Health and Human Services, and conducted by the OAI Study Investigators. Private funding partners include Merck Research Laboratories; Novartis Pharmaceuticals Corporation, GlaxoSmithKline; and Pfizer, Inc. Private sector funding for the OAI is managed by the Foundation for the National Institutes of Health. The authors wish to acknowledge CSC – IT Center for Science, Finland, for generous computational resources. We would like to acknowledge the strategic funding of the University of Oulu, Sigrid Juselius Foundation, Finland.
Copyright information: © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.