University of Oulu

Lukas Stappen, Alice Baird, Lukas Christ, Lea Schumann, Benjamin Sertolli, Eva-Maria Meßner, Erik Cambria, Guoying Zhao, and Björn W. Schuller. 2021. The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress. Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge. Association for Computing Machinery, New York, NY, USA, 5–14. DOI:

The MuSe 2021 multimodal sentiment analysis challenge : sentiment, emotion, physiological-emotion, and stress

Saved in:
Author: Stappen, Lukas1; Baird, Alice1; Christ, Lukas1;
Organizations: 1University of Augsburg, Augsburg, Germany
2University of Ulm, Ulm, Germany
3Nanyang Technological University, Singapore
4University of Oulu, Oulu, Finland
5Imperial College London, London, United Kingdom
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1.5 MB)
Persistent link:
Language: English
Published: Association for Computing Machinery, 2021
Publish Date: 2022-02-24


Multimodal Sentiment Analysis (MuSe) 2021 is a challenge focusing on the tasks of sentiment and emotion, as well as physiological-emotion and emotion-based stress recognition through more comprehensively integrating the audio-visual, language, and biological signal modalities. The purpose of MuSe 2021 is to bring together communities from different disciplines; mainly, the audio-visual emotion recognition community (signal-based), the sentiment analysis community (symbol-based), and the health informatics community. We present four distinct sub-challenges: MuSe-Wilder and MuSe-Stress which focus on continuous emotion (valence and arousal) prediction; MuSe-Sent, in which participants recognise five classes each for valence and arousal; and MuSe-Physio, in which the novel aspect of ‘physiological-emotion’ is to be predicted. For this year’s challenge, we utilise the MuSe-CaR dataset focusing on user-generated reviews and introduce the Ulm-TSST dataset, which displays people in stressful depositions. This paper also provides detail on the state-of-the-art feature sets extracted from these datasets for utilisation by our baseline model, a Long Short-Term Memory-Recurrent Neural Network. For each sub-challenge, a competitive baseline for participants is set; namely, on test, we report a Concordance Correlation Coefficient (CCC) of .4616 CCC for MuSe-Wilder; .5088 CCC for MuSe-Stress, and .4908 CCC for MuSe-Physio. For MuSe-Sent an F1 score of 32.82% is obtained.

see all

ISBN: 978-1-4503-8678-4
Pages: 5 - 14
DOI: 10.1145/3475957.348445
Host publication: 2nd Multimodal Sentiment Analysis Challenge and Workshop, MuSe 2021, held in conjunction with the ACM Multimedia 2021
Conference: Multimodal Sentiment Analysis Challenge and Workshop
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: This project has received funding from the European Union’s Horizon 2020 research and the DFG’s Reinhart Koselleck project No. 442218748 (AUDI0NOMOUS). We thank the sponsors of the Challenge, the BMW Group, and audEERING.
Copyright information: © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.