University of Oulu

Xiaopeng Hong, Wei Peng, Mehrtash Harandi, Ziheng Zhou, Matti Pietikäinen, and Guoying Zhao. 2019. Characterizing Subtle Facial Movements via Riemannian Manifold. ACM Trans. Multimedia Comput. Commun. Appl. 15, 3s, Article 94 (December 2019), 24 pages. DOI:

Characterizing subtle facial movements via Riemannian manifold

Saved in:
Author: Hong, Xiaopeng1,2; Peng, Wei2; Harandi, Mehrtash3,4;
Organizations: 1Xi’an Jiaotong University
2University of Oulu
3Monash University
5VeChain Foundation
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 3 MB)
Persistent link:
Language: English
Published: Association for Computing Machinery, 2019
Publish Date: 2020-04-23


Characterizing subtle facial movements from videos is one of the most intensive topics in computer vision research. It is, however, challenging, since (1) the intensity of subtle facial muscle movement is usually low, (2) the duration may be transient, and (3) datasets containing spontaneous subtle movements with reliable annotations are painful to obtain and often of small sizes.

This article is targeted at addressing these problems for characterizing subtle facial movements from both the aspects of motion elucidation and description. First, we propose an efficient method for elucidating hidden and repressed movements to make them easier to get noticed. We explore the feasibility of linearizing motion magnification and temporal interpolation, which is obscured by the architecture of existing methods. On this basis, we propose a consolidated framework, termed MOTEL, to expand temporal duration and amplify subtle facial movements simultaneously. Second, we make our contribution to dynamic description. One major challenge is to capture the intrinsic temporal variations caused by movements and omit extrinsic ones caused by different individuals and various environments. To diminish the influences of such extrinsic diversity, we propose the tangent delta descriptor to characterize the dynamics of short-term movements using the differences between points on the tangent spaces to the manifolds, rather than the points themselves. We then relax the trajectory-smooth assumption of the conventional manifold-based trajectory modeling methods and incorporate the tangent delta descriptor with the sequential inference approaches to cover the period of facial movements. The proposed motion modeling approach is validated by a series of experiments on publicly available datasets in the tasks of micro-expression recognition and visual speech recognition.

see all

Series: ACM transactions on multimedia computing, communications and applications
ISSN: 1551-6857
ISSN-E: 1551-6865
ISSN-L: 1551-6857
Volume: 15
Issue: 3S:
Article number: 94
DOI: 10.1145/3342227
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: We express deep gratitude to National Basic Research Program of China (Grant No. 2015CB351705), National Major Project (Grant No. 2017YFC0803905), and Natural Science Foundation of China under Contracts No. 61772419, No. 61572205, and No. 61601362, the Academy of Finland ICT 2023 (Project No. 313600), Infotech, Tekes Fidipro Program (Grant No. 1849/31/2015), Tekes project (Grant No. 3116/31/2017).
Academy of Finland Grant Number: 313600
Detailed Information: 313600 (Academy of Finland Funding decision)
Copyright information: © 2019 Association for Computing Machinery. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ACM Transactions on Multimedia Computing, Communications, and Applications,