University of Oulu

W. Peng, X. Hong and G. Zhao, "Adaptive Modality Distillation for Separable Multimodal Sentiment Analysis," in IEEE Intelligent Systems, doi: 10.1109/MIS.2021.3057757

Adaptive modality distillation for separable multimodal sentiment analysis

Saved in:
Author: Peng, Wei1; Hong, Xiaopeng2; Zhao, Guoying1
Organizations: 1University of Oulu
2Xian Jiaotong University
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.8 MB)
Persistent link: http://urn.fi/urn:nbn:fi-fe202104099805
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2021-04-09
Description:

Abstract

Multimodal sentiment analysis has increasingly attracted attention since with the arrival of complementary data streams, it has great potential to improve and go beyond unimodal sentiment analysis. In this paper, we present an efficient separable multimodal learning method to deal with the tasks with modality missing issue. In this method, the multimodal tensor is utilized to guide the evolution of each separated modality representation. To save the computational expense, Tucker decomposition is introduced, which leads to a general extension of the low-rank tensor fusion method with more modality interactions. The method, in turn, enhances our modality distillation processing. Comprehensive experiments on three popular multimodal sentiment analysis datasets, CMU-MOSI, POM, and IEMOCAP, show a superior performance especially when only partial modalities are available.

see all

Series: IEEE intelligent systems
ISSN: 1541-1672
ISSN-E: 1941-1294
ISSN-L: 1541-1672
Volume: Online First
Issue: Online First
Pages: 1 - 9
DOI: 10.1109/MIS.2021.3057757
OADOI: https://oadoi.org/10.1109/MIS.2021.3057757
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Subjects:
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists,or reuse of any copyrighted component of this work in other works.