University of Oulu

X. Li et al., "Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods," in IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 563-577, 1 Oct.-Dec. 2018. doi: 10.1109/TAFFC.2017.2667642

Towards reading hidden emotions : a comparative study of spontaneous micro-expression spotting and recognition methods

Saved in:
Author: Li, Xiaobai1; Hong, Xiaopeng1; Moilanen, Antti1;
Organizations: 1Center for Machine Vision and Signal Analysis, University of Oulu, Oulu 90014, Finland
2Department of Engineering Science, University of Oxford, Oxford OX1 3PA, United Kingdom
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 5.4 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2018
Publish Date: 2019-06-06


Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i) We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii) We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii) We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

see all

Series: IEEE transactions on affective computing
ISSN: 2371-9850
ISSN-E: 1949-3045
ISSN-L: 2371-9850
Volume: 9
Issue: 4
Pages: 563 - 577
DOI: 10.1109/TAFFC.2017.2667642
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
113 Computer and information sciences
Funding: This work was sponsored by the Academy of Finland, Infotech Oulu and Tekes Fidipro program. Guoying Zhao is the corresponding author of this paper.
Copyright information: © 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.