University of Oulu

Beddiar, D. R., Oussalah, M., Seppänen, T., & Jennane, R. (2022). ACapMed: Automatic Captioning for Medical Imaging. Applied Sciences, 12(21), 11092.

ACapMed : automatic captioning for medical imaging

Saved in:
Author: Beddiar, Djamila Romaissa1; Oussalah, Mourad1,2; Seppänen, Tapio1;
Organizations: 1Center for Machine Vision and Signal Analysis, University of Oulu, FI-90014 Oulu, Finland
2Faculty of Medicine, University of Oulu, FI-90014 Oulu, Finland
3IDP Laboratory—UMR CNRS 7013, University of Orleans, CEDEX 2, 45067 Orleans, France
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 9.9 MB)
Persistent link:
Language: English
Published: Multidisciplinary Digital Publishing Institute, 2022
Publish Date: 2022-11-22


Medical image captioning is a very challenging task that has been rarely addressed in the literature on natural image captioning. Some existing image captioning techniques exploit objects present in the image next to the visual features while generating descriptions. However, this is not possible for medical image captioning when one requires following clinician-like explanations in image content descriptions. Inspired by the preceding, this paper proposes using medical concepts associated with images, in accordance with their visual features, to generate new captions. Our end-to-end trainable network is composed of a semantic feature encoder based on a multi-label classifier to identify medical concepts related to images, a visual feature encoder, and an LSTM model for text generation. Beam search is employed to ensure the best selection of the next word for a given sequence of words based on the merged features of the medical image. We evaluated our proposal on the ImageCLEF medical captioning dataset, and the results demonstrate the effectiveness and efficiency of the developed approach.

see all

Series: Applied sciences
ISSN: 2076-3417
ISSN-E: 2076-3417
ISSN-L: 2076-3417
Volume: 12
Issue: 21
Article number: 11092
DOI: 10.3390/app122111092
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
114 Physical sciences
116 Chemical sciences
119 Other natural sciences
214 Mechanical engineering
Funding: This work is supported by the Academy of Finland Profi5 DigiHealth project (#326291), which is gratefully acknowledged.
Copyright information: © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (