Towards reading beyond faces for sparsity-aware 3D/4D affect recognition |
|
Author: | Behzad, Muzammil1; Vo, Nhat1; Li, Xiaobai1; |
Organizations: |
1Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland |
Format: | article |
Version: | published version |
Access: | open |
Online Access: | PDF Full Text (PDF, 2.7 MB) |
Persistent link: | http://urn.fi/urn:nbn:fi-fe2021100649452 |
Language: | English |
Published: |
Elsevier,
2021
|
Publish Date: | 2021-10-06 |
Description: |
AbstractIn this paper, we present a sparsity-aware deep network for automatic 3D/4D facial expression recognition (FER). We first propose a novel augmentation method to combat the data limitation problem for deep learning, specifically given 3D/4D face meshes. This is achieved by projecting the input data into RGB and depth map images and then iteratively performing randomized channel concatenation. Encoded in the given 3D landmarks, we also introduce an effective way to capture the facial muscle movements from three orthogonal plans (TOP), the TOP-landmarks over multi-views. Importantly, we then present a sparsity-aware deep network to compute the sparse representations of convolutional features over multi-views. This is not only effective for a higher recognition accuracy but also computationally convenient. For training, the TOP-landmarks and sparse representations are used to train a long short-term memory (LSTM) network for 4D data, and a pre-trained network for 3D data. The refined predictions are achieved when the learned features collaborate over multi-views. Extensive experimental results achieved on the Bosphorus, BU-3DFE, BU-4DFE and BP4D-Spontaneous datasets show the significance of our method over the state-of-the-art methods and demonstrate its effectiveness by reaching a promising accuracy of 99.69% on BU-4DFE for 4D FER. see all
|
Series: |
Neurocomputing |
ISSN: | 0925-2312 |
ISSN-E: | 1872-8286 |
ISSN-L: | 0925-2312 |
Volume: | 458 |
Pages: | 297 - 307 |
DOI: | 10.1016/j.neucom.2021.06.023 |
OADOI: | https://oadoi.org/10.1016/j.neucom.2021.06.023 |
Type of Publication: |
A1 Journal article – refereed |
Field of Science: |
113 Computer and information sciences |
Subjects: | |
Funding: |
This work was supported by Infotech Oulu, the Academy of Finland for project MiGA (grant 316765), project 6 + E (grant 323287), and ICT 2023 project (grant 328115). As well, the financial supports
from Riitta ja Jorma J. Takanen Foundation and Tauno Tönning Foundation are acknowledged. Lastly, the authors wish to acknowledge CSC – IT Center for Science, Finland, for computational
resources. |
Academy of Finland Grant Number: |
316765 323287 328115 |
Detailed Information: |
316765 (Academy of Finland Funding decision) 323287 (Academy of Finland Funding decision) 328115 (Academy of Finland Funding decision) |
Copyright information: |
© 2021 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). |
https://creativecommons.org/licenses/by/4.0/ |