University of Oulu

C. Guo, J. Liang, G. Zhan, Z. Liu, M. Pietikäinen and L. Liu, "Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition," in IEEE Access, vol. 7, pp. 174517-174530, 2019. doi: 10.1109/ACCESS.2019.2942358

Extended local binary patterns for efficient and robust spontaneous facial micro-expression recognition

Saved in:
Author: Guo, Chengyu1; Liang, Jingyun1; Zhan, Geng2;
Organizations: 1College of Systems Engineering, National University of Defense Technology, Changsha 410073, China
2School of Electrical and Information Engineering, The University of Sydney, Camperdown, NSW 2006, Australia
3Center for Machine Vision and Signal Analysis, University of Oulu, 90570 Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 2.6 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2019
Publish Date: 2020-01-13


Facial Micro-Expressions (MEs) are spontaneous, involuntary facial movements when a person experiences an emotion but deliberately or unconsciously attempts to conceal his or her genuine emotions. Recently, ME recognition has attracted increasing attention due to its potential applications such as clinical diagnosis, business negotiation, interrogations, and security. However, it is expensive to build large scale ME datasets, mainly due to the difficulty of inducing spontaneous MEs. This limits the application of deep learning techniques which require lots of training data. In this paper, we propose a simple, efficient yet robust descriptor called Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) for ME recognition. ELBPTOP consists of three complementary binary descriptors: LBPTOP and two novel ones Radial Difference LBPTOP (RDLBPTOP) and Angular Difference LBPTOP (ADLBPTOP), which explore the local second order information along the radial and angular directions contained in ME video sequences. ELBPTOP is a novel ME descriptor inspired by unique and subtle facial movements. It is computationally efficient and only marginally increases the cost of computing LBPTOP, yet is extremely effective for ME recognition. In addition, by firstly introducing Whitened Principal Component Analysis (WPCA) to ME recognition, we can further obtain more compact and discriminative feature representations, then achieve significantly computational savings. Extensive experimental evaluation on three popular spontaneous ME datasets SMIC, CASME II and SAMM show that our proposed ELBPTOP approach significantly outperforms the previous state-of-the-art on all three single evaluated datasets and achieves promising results on cross-database recognition. Our code will be made available.

see all

Series: IEEE access
ISSN: 2169-3536
ISSN-E: 2169-3536
ISSN-L: 2169-3536
Volume: 7
Pages: 174517 - 174530
DOI: 10.1109/ACCESS.2019.2942358
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was supported by the National Natural Science Foundation of China under Grant 61872379.
Copyright information: © The Authors 2019. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see