University of Oulu

M. Behzad, X. Li and G. Zhao, "Disentangling 3D/4D Facial Affect Recognition With Faster Multi-View Transformer," in IEEE Signal Processing Letters, vol. 28, pp. 1913-1917, 2021, doi: 10.1109/LSP.2021.3111576

Disentangling 3D/4D facial affect recognition with faster multi-view transformer

Saved in:
Author: Behzad, Muzammil1; Li, Xiaobai1; Zhao, Guoying1
Organizations: 1Center for Machine Vision and Signal Analysis, University of Oulu, Oulu 90570, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 1.3 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2021-10-06


In this paper, we propose MiT: a novel multi-view transformer model 1 for 3D/4D facial affect recognition. MiT incorporates patch and position embeddings from various patches of multi-views and uses them for learning various facial muscle movements to showcase an effective recognition performance. We also propose a multi-view loss function that is not only gradient-friendly, and hence speeds up the gradient computation during back-propagation, but it also leverages the correlation associated with the underlying facial patterns among multi-views. Additionally, we offer multi-view weights that are trainable and learnable, and help substantially in training. Finally, we equip our model with distributed performance for faster learning and computational convenience. With the help of extensive experiments, we show that our model outperform the existing methods on widely-used datasets for 3D/4D FER.

see all

Series: IEEE signal processing letters
ISSN: 1070-9908
ISSN-E: 1558-2361
ISSN-L: 1070-9908
Volume: 28
Pages: 1913 - 1917
DOI: 10.1109/LSP.2021.3111577
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was supported in part by Infotech Oulu and the Academy of Finland through project MiGA under Grant 316765, project 6 + E under Grant 323287, and ICT 2023 project under Grant 328115; in part by the Riitta ja Jorma J. Takanen Foundation; and in part by the Tauno Tönning Foundation.
Academy of Finland Grant Number: 316765
Detailed Information: 316765 (Academy of Finland Funding decision)
323287 (Academy of Finland Funding decision)
328115 (Academy of Finland Funding decision)
Copyright information: © 2021 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see