University of Oulu

M. Behzad, N. Vo, X. Li and G. Zhao, "Landmarks-assisted Collaborative Deep Framework for Automatic 4D Facial Expression Recognition," 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina, 2020, pp. 1-5, doi: 10.1109/FG47880.2020.00023

Landmarks-assisted collaborative deep framework for automatic 4D facial expression recognition

Saved in:
Author: Behzad, Muzammil1; Vo, Nhat1; Li, Xiaobai1;
Organizations: 1Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 6.7 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2020
Publish Date: 2021-03-03


We propose a novel landmarks-assisted collaborative end-to-end deep framework for 4D facial expression recognition (FER). Using 4D face scan data, we calculate its various geometrical images, and afterwards use rank pooling to generate their dynamic images encapsulating important facial muscle movements over time. As well, the given 3D landmarks are projected on a 2D plane as binary images and convolutional layers are used to extract sequences of feature vectors for every landmark video. During the training stage, the dynamic images are used to train an end-to-end deep network, while the feature vectors of landmark images are used train a long short-term memory (LSTM) network. The finally improved set of expression predictions are obtained when the dynamic and landmark images collaborate over multi-views using the proposed deep framework. Performance results obtained from extensive experimentation on the widely-adopted BU-4DFE database under globally used settings prove that our proposed collaborative framework outperforms the state-of-the-art 4D FER methods and reach a promising classification accuracy of 96.7% demonstrating its effectiveness.

see all

ISBN: 978-1-7281-3079-8
ISBN Print: 978-1-7281-3080-4
Pages: 1 - 5
Article number: 9320291
DOI: 10.1109/FG47880.2020.00023
Host publication: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)
Conference: IEEE International Conference on Automatic Face and Gesture Recognition
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: This work was supported by Infotech Oulu, the National Natural Science Foundation of China (No. 61772419), Tekes Fidipro Program (No. 1849/31/2015), Business Finland Project (No. 3116/31/2017), and Academy of Finland. As well, the authors wish to acknowledge CSC IT Center for Science, Finland, for computational resources.
Copyright information: © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.