University of Oulu

H. Shi, W. Peng, H. Chen, X. Liu and G. Zhao, "Multiscale 3D-Shift Graph Convolution Network for Emotion Recognition From Human Actions," in IEEE Intelligent Systems, vol. 37, no. 4, pp. 103-110, 1 July-Aug. 2022, doi: 10.1109/MIS.2022.3147585.

Multiscale 3D-shift graph convolution network for emotion recognition from human actions

Saved in:
Author: Shi, Henglin1; Peng, Wei1; Chen, Haoyu1;
Organizations: 1University of Oulu, 90570, Oulu, Finland
2Lappeenranta-Lahti University of Technology LUT, 53850, Lappeenranta, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.9 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2023-01-26


Emotion recognition from body gestures is challenging since similar emotions can be expressed by arbitrary spatial configurations of joints, which results in relying on modeling spatial-temporal patterns from a more global level. However, most recent powerful graph convolution networks (GCNs) separate the spatial and temporal modeling into isolated processes, where GCN models spatial interactions using partially fixed adjacent matrices and 1D convolution captures temporal dynamics, which is insufficient for emotion recognition. In this work, we propose the 3D-Shift GCN, which enables interactions of joints within a spatial-temporal volume for global feature extraction. Besides, we further develop a multiscale architecture, the MS-Shift GCN, to fuse features captured under different temporal ranges for modeling richer dynamics. After conducting evaluation on two regular action recognition benchmarks and two gesture based emotion recognition datasets, the results show that the proposed method outperforms several state-of-the-art methods.

see all

Series: IEEE intelligent systems
ISSN: 1541-1672
ISSN-E: 1941-1294
ISSN-L: 1541-1672
Volume: 37
Issue: 4
Pages: 103 - 110
DOI: 10.1109/MIS.2022.3147585
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was supported in part by the Academy of Finland for ICT 2023 project under Grant 328115, in part by the project MiGA under Grant 316765, and in part by the Infotech Oulu. The authors would like to thank CSC-IT Center for Science, Finland, for computational resources.
Academy of Finland Grant Number: 328115
Detailed Information: 328115 (Academy of Finland Funding decision)
316765 (Academy of Finland Funding decision)
Copyright information: © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.