University of Oulu

D. Kollias, M. A. Nicolaou, I. Kotsia, G. Zhao and S. Zafeiriou, "Recognition of Affect in the Wild Using Deep Neural Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp. 1972-1979. doi: 10.1109/CVPRW.2017.247

Recognition of affect in the wild using deep neural networks

Saved in:
Author: Kollias, Dimitrios1; Nicolaou, Mihalis A.2; Kotsia, Irene3,4;
Organizations: 1Department of Computing, Imperial College London, UK
2Department of Computing, Goldsmiths, University of London, UK
3School of Science and Technology, International Hellenic University, Greece
4Department of Computer Science, Middlesex University, UK
5Center for Machine Vision and Signal Analysis, University of Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 1 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2017
Publish Date: 2019-02-27


In this paper we utilize the first large-scale "in-the-wild" (Aff-Wild) database, which is annotated in terms of the valence-arousal dimensions, to train and test an end-to-end deep neural architecture for the estimation of continuous emotion dimensions based on visual cues. The proposed architecture is based on jointly training convolutional (CNN) and recurrent neural network (RNN) layers, thus exploiting both the invariant properties of convolutional features, while also modelling temporal dynamics that arise in human behaviour via the recurrent layers. Various pre-trained networks are used as starting structures which are subsequently appropriately fine-tuned to the Aff-Wild database. Obtained results show premise for the utilization of deep architectures for the visual analysis of human behaviour in terms of continuous emotion dimensions and analysis of different types of affect.

see all

Series: IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops
ISSN: 2160-7508
ISSN-E: 2160-7516
ISSN-L: 2160-7508
ISBN Print: 9781538607336
Pages: 1972 - 1979
DOI: 10.1109/CVPRW.2017.247
Host publication: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, Hawaii 21-26 July 2017
Conference: IEEE Conference on Computer Vision and Pattern Recognition
Type of Publication: A4 Article in conference proceedings
Field of Science: 113 Computer and information sciences
Funding: The work of Stefanos Zafeiriou has been partially funded by the FiDiPro program of Tekes (project number: 1849/31/2015). The work of Dimitris Kollias was funded by a Teaching Fellowship of Imperial College London. We would like also to acknowledge the contribution of the Youtube users that gave us the permission to use their videos (especially Zalzar and Eddie from The1stTake).
Copyright information: © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.