University of Oulu

S. Sieranoja, M. Sahidullah, T. Kinnunen, J. Komulainen and A. Hadid, "Audiovisual Synchrony Detection with Optimized Audio Features," 2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP), Shenzhen, 2018, pp. 377-381,

Audiovisual synchrony detection with optimized audio features

Saved in:
Author: Sieranoja, Sami1; Kinnunen, Tomi1; Komulainen, Jukka2;
Organizations: 1School of Computing, University of Eastern Finland, Joensuu, Finland
2Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 2.6 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2018
Publish Date: 2020-04-14


Audiovisual speech synchrony detection is an important part of talking-face verification systems. Prior work has primarily focused on visual features and joint-space models, while standard mel-frequency cepstral coefficients (MFCCs) have been commonly used to present speech. We focus more closely on audio by studying the impact of context window length for delta feature computation and comparing MFCCs with simpler energy-based features in lip-sync detection. We select state-of-the-art hand-crafted lip-sync visual features, space-time auto-correlation of gradients (STACOG), and canonical correlation analysis (CCA), for joint-space modeling. To enhance joint space modeling, we adopt deep CCA (DCCA), a nonlinear extension of CCA. Our results on the XM2VTS data indicate substantially enhanced audiovisual speech synchrony detection, with an equal error rate (EER) of 3.68%. Further analysis reveals that failed lip region localization and beardedness of the subjects constitutes most of the errors. Thus, the lip motion description is the bottleneck, while the use of novel audio features or joint-modeling techniques is unlikely to boost lip-sync detection accuracy further.

see all

ISBN: 978-1-5386-6396-7
ISBN Print: 978-1-5386-6397-4
Pages: 377 - 381
DOI: 10.1109/SIPROCESS.2018.8600424
Host publication: 2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP)
Conference: International Conference on Signal and Image Processing
Type of Publication: A4 Article in conference proceedings
Field of Science: 213 Electronic, automation and communications engineering, electronics
113 Computer and information sciences
Funding: The project was partially funded by Academy of Finland and the Finnish Foundation for Technology Promotion.
Copyright information: © 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.