University of Oulu

Muzammil Behzad, Nhat Vo, Xiaobai Li, Guoying Zhao, Towards Reading Beyond Faces for Sparsity-aware 3D/4D Affect Recognition, Neurocomputing, Volume 458, 2021, Pages 297-307, ISSN 0925-2312,

Towards reading beyond faces for sparsity-aware 3D/4D affect recognition

Saved in:
Author: Behzad, Muzammil1; Vo, Nhat1; Li, Xiaobai1;
Organizations: 1Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 2.7 MB)
Persistent link:
Language: English
Published: Elsevier, 2021
Publish Date: 2021-10-06


In this paper, we present a sparsity-aware deep network for automatic 3D/4D facial expression recognition (FER). We first propose a novel augmentation method to combat the data limitation problem for deep learning, specifically given 3D/4D face meshes. This is achieved by projecting the input data into RGB and depth map images and then iteratively performing randomized channel concatenation. Encoded in the given 3D landmarks, we also introduce an effective way to capture the facial muscle movements from three orthogonal plans (TOP), the TOP-landmarks over multi-views. Importantly, we then present a sparsity-aware deep network to compute the sparse representations of convolutional features over multi-views. This is not only effective for a higher recognition accuracy but also computationally convenient. For training, the TOP-landmarks and sparse representations are used to train a long short-term memory (LSTM) network for 4D data, and a pre-trained network for 3D data. The refined predictions are achieved when the learned features collaborate over multi-views. Extensive experimental results achieved on the Bosphorus, BU-3DFE, BU-4DFE and BP4D-Spontaneous datasets show the significance of our method over the state-of-the-art methods and demonstrate its effectiveness by reaching a promising accuracy of 99.69% on BU-4DFE for 4D FER.

see all

Series: Neurocomputing
ISSN: 0925-2312
ISSN-E: 1872-8286
ISSN-L: 0925-2312
Volume: 458
Pages: 297 - 307
DOI: 10.1016/j.neucom.2021.06.023
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was supported by Infotech Oulu, the Academy of Finland for project MiGA (grant 316765), project 6 + E (grant 323287), and ICT 2023 project (grant 328115). As well, the financial supports from Riitta ja Jorma J. Takanen Foundation and Tauno Tönning Foundation are acknowledged. Lastly, the authors wish to acknowledge CSC – IT Center for Science, Finland, for computational resources.
Academy of Finland Grant Number: 316765
Detailed Information: 316765 (Academy of Finland Funding decision)
323287 (Academy of Finland Funding decision)
328115 (Academy of Finland Funding decision)
Copyright information: © 2021 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (