University of Oulu

D. Huang et al., "Spatio-Temporal Pain Estimation Network with Measuring Pseudo Heart Rate Gain," in IEEE Transactions on Multimedia, doi: 10.1109/TMM.2021.3096080

Spatio-temporal pain estimation network with measuring pseudo heart rate gain

Saved in:
Author: Huang, Dong1; Feng, Xiaoyi1; Zhang, Haixi2;
Organizations: 1School of Electronics and Information, Northwestern Polytechnical University, Xi’an 710129, China
2College of Information Engineering, Northwest A&F University, Yangling 712100, China
3Center for Machine Vision and Signal Analysis, University of Oulu, 90014 Oulu, Finland
4School of Information Science and Technology, Northwest University, Xi’an 710127, China
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 11.7 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2022-01-04


Pain is a significant indicator that shows people are suffering from an unwell experience and its automatic estimation has attracted much interest in recent years. Of late, most estimation methods are designed to capture the dynamic pain information from visual signals while a few physiological-signal based methods can provide extra potential cues to analyze the pain more accurately. However, it is still challenging to capture the physiological data from patients as it requires contact devices and patients’ cooperation. In this paper, we propose to leverage the pseudo physiological information by generating new modal data from the original visual videos and jointly estimating the pain by an end-to-end network. To extract the representations from bi-modal data, we design a spatio-temporal pain estimation network, which employs a dual-branch framework for extracting pain-aware visual and pseudo physiological features separately and fuses the features in a probabilistic way. The inherent vital sign, i.e., heart rate gain (HRG), from pseudo physiological information can be utilized as an auxiliary signal and integrated with the visual pain estimation framework. Moreover, specially-designed 3D convolution filters and attention structures are employed to extract spatio-temporal features for both branches. To use the HRG as an auxiliary way for pain estimation, we propose a probabilistic inference model by jointly considering the visual branch and physiological branch, which makes our model estimate the pain comprehensively. Experiments on two publicly-available datasets show the effectiveness of introducing the pseudo modality, and the proposed method can outperform the state-of-the-art methods.

see all

Series: IEEE transactions on multimedia
ISSN: 1520-9210
ISSN-E: 1941-0077
ISSN-L: 1520-9210
Volume: In press
DOI: 10.1109/TMM.2021.3096080
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work is partly supported by the Key Research and Development Program of Shaanxi (Program Nos. 2021ZDLGY15-01, 2021ZDLGY09-04, 2021GY-004 and 2020GY-050), the National Natural Science Foundation of China (Grant No. 61772419), Academy of Finland for project MiGA (Grant No. 316765) , and ICT 2023 project (Grant No. 328115), Infotech Oulu.
Academy of Finland Grant Number: 316765
Detailed Information: 316765 (Academy of Finland Funding decision)
328115 (Academy of Finland Funding decision)
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.