University of Oulu

C. Sheng, X. Zhu, H. Xu, M. Pietikäinen and L. Liu, "Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading," in IEEE Transactions on Multimedia, vol. 24, pp. 3545-3557, 2022, doi: 10.1109/TMM.2021.3102433

Adaptive semantic-spatio-temporal graph convolutional network for lip reading

Saved in:
Author: Sheng, Changchong1; Zhu, Xinzhong2,2; Xu, Huiying2,3;
Organizations: 1Center for Machine Vision and Signal Analysis, University of Oulu, Finland
2College of Mathematics, Physics and Information Engineering, Zhejiang Normal University, Jinhua 321004, China
3Research Institute of Ningbo Cixing Co. Ltd, China
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 2.8 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2022-10-06


The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.

see all

Series: IEEE transactions on multimedia
ISSN: 1520-9210
ISSN-E: 1941-0077
ISSN-L: 1520-9210
Volume: 24
Pages: 3545 - 3557
DOI: 10.1109/tmm.2021.3102433
Type of Publication: A1 Journal article – refereed
Field of Science: 113 Computer and information sciences
Funding: This work was partially supported by the Academy of Finland under grant 331883, Outstanding Talents of “Ten Thousand Talents Plan” in Zhejiang Province (project no. 2018R51001), and the Natural Science Foundation of China (project no. 61976196).
Academy of Finland Grant Number: 331883
Detailed Information: 331883 (Academy of Finland Funding decision)
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.