University of Oulu

Tavakolian, M., Hadid, A. (2019) A Spatiotemporal Convolutional Neural Network for Automatic Pain Intensity Estimation from Facial Dynamics. International journal of computer vision, 127 (10), 1413-1425. doi:10.1007/s11263-019-01191-3

A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics

Saved in:
Author: Tavakolian, Mohammad1; Hadid, Abdenour1
Organizations: 1Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 2.1 MB)
Persistent link:
Language: English
Published: Springer Nature, 2019
Publish Date: 2019-10-08


Devising computational models for detecting abnormalities reflective of diseases from facial structures is a novel and emerging field of research in automatic face analysis. In this paper, we focus on automatic pain intensity estimation from faces. This has a paramount potential diagnosis values in healthcare applications. In this context, we present a novel 3D deep model for dynamic spatiotemporal representation of faces in videos. Using several convolutional layers with diverse temporal depths, our proposed model captures a wide range of spatiotemporal variations in the faces. Moreover, we introduce a cross-architecture knowledge transfer technique for training 3D convolutional neural networks using a pre-trained 2D architecture. This strategy is a practical approach for training 3D models, especially when the size of the database is relatively small. Our extensive experiments and analysis on two benchmarking and publicly available databases, namely the UNBC-McMaster shoulder pain and the BioVid, clearly show that our proposed method consistently outperforms many state-of-the-art methods in automatic pain intensity estimation.

see all

Series: International journal of computer vision
ISSN: 0920-5691
ISSN-E: 1573-1405
ISSN-L: 0920-5691
Volume: 127
Issue: 10
Pages: 1413 - 1425
DOI: 10.1007/s11263-019-01191-3
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: Open access funding provided by University of Oulu including Oulu University Hospital. The financial support of the Academy of Finland, Infotech Oulu, Nokia Foundation, and Tauno Tönning Foundation is acknowledged.
Copyright information: © The Author(s) 2019. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.