University of Oulu

Chen, W., Wang, W., Liu, L., & Lew, M. S. (2021). New Ideas and Trends in Deep Multimodal Content Understanding: A Review. Neurocomputing, 426, 195–215.

New ideas and trends in deep multimodal content understanding : a review

Saved in:
Author: Chen, Wei1; Wang, Weiping2; Liu, Li2,3;
Organizations: 1LIACS, Leiden University, Leiden, 2333 CA, The Netherlands
2College of Systems Engineering, NUDT, Changsha, 410073, China
3Center for Machine Vision and Signal Analysis, University of Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 3.1 MB)
Persistent link:
Language: English
Published: Elsevier, 2021
Publish Date: 2021-04-19


The focus of this survey is on the analysis of two modalities of multimodal deep learning: image and text. Unlike classic reviews of deep learning where monomodal image classifiers such as VGG, ResNet and Inception module are central topics, this paper will examine recent multimodal deep models and structures, including auto-encoders, generative adversarial nets and their variants. These models go beyond the simple image classifiers in which they can do uni-directional (e.g. image captioning, image generation) and bi-directional (e.g. cross-modal retrieval, visual question answering) multimodal tasks. Besides, we analyze two aspects of the challenge in terms of better content understanding in deep multimodal applications. We then introduce current ideas and trends in deep multimodal feature learning, such as feature embedding approaches and objective function design, which are crucial in overcoming the aforementioned challenges. Finally, we include several promising directions for future research.

see all

Series: Neurocomputing
ISSN: 0925-2312
ISSN-E: 1872-8286
ISSN-L: 0925-2312
Volume: 426
Pages: 195 - 215
DOI: 10.1016/j.neucom.2020.10.042
Type of Publication: A2 Review article in a scientific journal
Field of Science: 113 Computer and information sciences
Funding: This work was supported by LIACS MediaLab at Leiden University and China Scholarship Council (CSC No. 201703170183). We appreciate the helpful editing work from Erwin Bakker.
Copyright information: © 2020 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (