University of Oulu

Mohammad Tavakolian, Miguel Bordallo Lopez, Li Liu, Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation, Pattern Recognition Letters, Volume 140, 2020, Pages 26-33, ISSN 0167-8655,

Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation

Saved in:
Author: Tavakolian, Mohammad1; Bordallo Lopez, Miguel2; Liu, Li1,3
Organizations: 1Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland
2VTT Technical Research Centre of Finland Ltd, Oulu, Finland
3National University of Defense Technology, China
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 5.8 MB)
Persistent link:
Language: English
Published: Elsevier, 2020
Publish Date: 2022-09-19


Recently, automatic pain assessment technology, in particular automatically detecting pain from facial expressions, has been developed to improve the quality of pain management, and has attracted increasing attention. In this paper, we propose self-supervised learning for automatic yet efficient pain assessment, in order to reduce the cost of collecting large amount of labeled data. To achieve this, we introduce a novel similarity function to learn generalized representations using a Siamese network in the pretext task. The learned representations are finetuned in the downstream task of pain intensity estimation. To make the method computationally efficient, we propose Statistical Spatiotemporal Distillation (SSD) to encode the spatiotemporal variations underlying the facial video into a single RGB image, enabling the use of less complex 2D deep models for video representation. Experiments on two publicly available pain datasets and cross-dataset evaluation demonstrate promising results, showing the good generalization ability of the learned representations.

see all

Series: Pattern recognition letters
ISSN: 0167-8655
ISSN-E: 1872-7344
ISSN-L: 0167-8655
Volume: 140
Pages: 26 - 33
DOI: 10.1016/j.patrec.2020.09.012
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: We would like to acknowledge the financial support of the Academy of Finland (No. 331883), Infotech Oulu, Tauno Tönning, Nokia, and KAUTE foundations.
Academy of Finland Grant Number: 331883
Detailed Information: 331883 (Academy of Finland Funding decision)
Copyright information: © 2020 Elsevier B.V. All rights reserved. This manuscript version is made available under the CC-BY-NC-ND 4.0 license